Design, develop, and deploy scalable data infrastructure on Databricks for the Department of State’s data lake, leading architecture, governance, and CI/CD pipelines.
Responsibilities
Design and implement scalable data pipelines using Databricks and PySpark
Architect and enforce Medallion Architecture for data quality
Develop and maintain custom Python libraries and modules
Integrate data ingestion from RESTful APIs
Automate deployments with Azure DevOps and Databricks Asset Bundles
Lead the technical design, delivery, and evolution of MSS’s Weather Data Lake, managing data pipelines, governance, and cloud infrastructure on Microsoft Azure.
Compensation
This role will start off on a 2 year direct contract
Responsibilities
Lead platform architecture and technical direction
Set the technical direction for MSS’s Weather Data Lake
Work with domain specialists and engineering teams to translate meteorological and climate data requirements into technical designs and delivery plans
Define clear boundaries and interfaces between ingestion systems, the data lake, and downstream services
Document architectural decisions and build alignment on technical priorities, dependencies, and trade-offs
Design, develop, and maintain scalable data pipelines and Big Data solutions on Microsoft Azure and related platforms.
Responsibilities
Design, develop, and maintain scalable data pipelines using PySpark, Spark SQL, Python, SQL, and Azure Synapse Analytics.
Build and optimize ETL/ELT workflows for large-volume structured, semi-structured, and unstructured data processing.
Develop and support Azure Synapse pipelines, notebooks, SQL scripts, stored procedures, views, and data transformation logic.
Work with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Databricks, Azure SQL Database, and related Azure data services.
Develop and support Big Data solutions using technologies such as Apache Spark, Hadoop, HDFS, Hive, Sqoop, Kafka, HBase, and related distributed data processing components.
Power BI Solution Architect responsible for defining, designing, and governing enterprise‑wide Power BI and analytics solutions, providing architectural oversight, stakeholder leadership, and mentorship across BI and data teams.
Responsibilities
Lead the design and implementation of end‑to‑end Power BI solutions
Define enterprise Power BI architecture standards and governance frameworks
Design scalable and performant data models
Guide architectural decisions for data integration and performance optimization
Establish and enforce Power BI governance, tenant administration, and security standards
Requirements
B. Tech (Computer Science / Information Technology)
Senior Platform Engineer focused on data, cloud platform, and networking to build reliable, secure, and scalable data infrastructure on Azure.
Compensation
This position pays between $101,600 – $152,400 based on experience
Responsibilities
Design, build, test, deploy, monitor, and continuously improve scalable data pipelines using Azure Data Factory, Databricks, Apache Spark, and Python.
Implement Delta Lake patterns on Azure Data Lake Storage including table design, schema evolution, Z-ordering, VACUUM, and partition strategies.
Build and maintain event-driven integrations using Azure Service Bus, Event Hub, and Azure Functions.
Provision and manage Azure infrastructure using infrastructure as code (Terraform, Bicep, or ARM templates).
Design and operate Azure-native services including Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2, Microsoft Fabric, Azure Key Vault, and Azure Monitor.
Requirements
Bachelor's degree in Computer Science, Engineering, Information Systems, or equivalent practical experience
Design, develop, test, deploy, and maintain scalable data pipelines using Databricks and related technologies.
Responsibilities
Design, develop, test, deploy, optimize, and support complex, scalable, production-grade data pipelines using Databricks and related data engineering technologies.
Develop robust ETL/ELT solutions to ingest, transform, integrate, and deliver data across cloud and enterprise environments.
Design reusable and maintainable data integration patterns leveraging Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, APIs, and other appropriate technologies.
Design and maintain efficient data models, data structures, and storage solutions.
Implement data quality, validation, reconciliation, monitoring, and error-handling processes to ensure reliable and accurate data delivery.
Requirements
Bachelor's degree in computer science, Information Systems, Engineering, Data Engineering, or a related field, or equivalent professional experience
We are looking for an experienced Data Engineer to join a project focused on building and developing modern cloud-based data solutions using Microsoft Azure, Azure Data Factory, and Snowflake.
Design, build, and optimize data pipelines using Databricks, PySpark, Python, and SQL; develop scalable ETL/ELT solutions and Lakehouse architectures; implement solutions using Delta Lake, Unity Catalog, Databricks Workflows, and Structured Streaming; develop and support Azure-based data platforms leveraging ADF and ADLS Gen2; optimize Spark workloads for performance, scalability, and cost efficiency; implement CI/CD, DataOps, and engineering best practices; collaborate with business and technical stakeholders to deliver high-quality data solutions; support data governance, quality, and security initiatives; mentor junior engineers and contribute to capability development.
Responsibilities
Design, build, and optimize data pipelines using Databricks, PySpark, Python, and SQL.
Develop scalable ETL/ELT solutions and Lakehouse architectures.
Implement solutions using Delta Lake, Unity Catalog, Databricks Workflows, and Structured Streaming.
Develop and support Azure-based data platforms leveraging ADF and ADLS Gen2.
Optimize Spark workloads for performance, scalability, and cost efficiency.
Requirements
Nice to have: Databricks Certified Data Engineer Associate
Nice to have: Microsoft Azure Data Engineer Associate (DP-203)
Generous referral bonuses, offering up to PLN6,000 for referring specialists. Additional revenue sharing opportunities for initiating partnerships with new clients.
Responsibilities
Design, build and maintain batch and incremental ETL/ELT pipelines from source systems to Snowflake.
Develop and orchestrate data pipelines using Azure Data Factory, including linked services, datasets, data flows, triggers and reusable parameterised pipelines.
Design and maintain data models in Snowflake, covering staging, core and data mart layers.
Manage Snowflake warehouses, schemas, roles and data-loading mechanisms such as COPY INTO, Snowpipe and stages.
Write, optimise and maintain complex SQL queries for data transformations, reporting and data quality validation.
Senior Databricks EngineerNewKoniag Government Services, LLC · Remote, US
Staff Data EngineerNewGovernment of Singapore – Public Service (Careers@Gov) · Singapore
Senior Azure Data EngineerCognizant
Senior Data ScientistCapgemini · Bogota, Colombia
Director-GDC-TechnicalKPMG Global Services · Bangalore, India
Sr. Data EngineerGM Financial IO · Monterrey, Mexico