Back to search

Azure Data Engineer

Collect, store, process, and analyze large datasets to build and implement extract, transform, and load (ETL) processes, leading migration of data pipelines from Synapse to Databricks.

  • Cary, North Carolina, United States
  • Contract
  • On-site
  • $50–$52 per hour
  • Posted Nov 13, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software

Responsibilities

  • Collect, store, process, and analyze large datasets to build and implement extract, transform, and load (ETL) processes
  • Lead migration of data pipelines from Synapse to Databricks
  • Architect and implement batch and real-time data pipelines using PySpark, Databricks, and Event Hubs
  • Establish data engineering best practices and mentor junior engineers
  • Drive CI/CD automation with Azure DevOps (AzDO) and Git
  • Ensure data quality, governance, and compliance using Collibra
  • Collaborate with business and technical teams; leverage insurance domain knowledge where applicable
  • Develop reusable frameworks to reduce development effort and ensure cost savings for projects
  • Develop quality code with performance optimizations implemented during the development stage
  • Learn new technologies and adapt to emerging cloud technologies
  • Work with globally distributed teams to drive project delivery and recommend development and performance improvements

Requirements

  • Experience in a insurance

Nice to have: Collibra, Azure/Databricks certifications, Azure Functions, Node.js

Skills

Microsoft AzureAzure DatabricksSynapsePySparkEvent HubsSQLCI/CDSparkScalaPythonData LakeData FactoryKey VaultAzure DevOpsGitCollibraDatabricksSynapsePySparkEvent HubsADLSAzure Data FactoryCosmos DBDelta LakeHiveHDFSSqoopKafkaKerberosImpalaCDP 7.xBitbucket

More jobs nearby

Full description

Job Title: Azure Data Engineer Location: Cary, NC (Local Candidates Only) Duration: 6 Months Interview Mode: Face-to-Face Only Salary Range: $50.00 - $52.00/Hour on W2 (Without Benefits). Applicants must be willing to work on W2 Key Responsibilities: Overall Experience Required is 8–10 Years. Must Have 7+ years of Experience in Microsoft Azure. Collect, store, process, and analyze large datasets to build and implement extract, transform, and load (ETL) processes. Lead migration of data pipelines from Synapse to Databricks. Architect and implement batch and real-time data pipelines using PySpark, Databricks, and Event Hubs. Establish data engineering best practices and mentor junior engineers. Drive CI/CD automation with Azure DevOps (AzDO) and Git. Ensure data quality, governance, and compliance using Collibra. Collaborate with business and technical teams; leverage insurance domain knowledge where applicable. Develop reusable frameworks to reduce development effort and ensure cost savings for projects. Develop quality code with performance optimizations implemented during the development stage. Learn new technologies and adapt to emerging cloud technologies. Work with globally distributed teams to drive project delivery and recommend development and performance improvements. Essential Business Experience and Technical Skills: 7+ years of experience in data engineering, with 3+ years in a lead/senior role. Strong expertise in Azure Databricks, Synapse, and PySpark. Proven experience with real-time streaming technologies, including Event Hubs, Structured Streaming, and Kafka. Deep knowledge of the Azure ecosystem, including Data Lake, Data Factory, and Key Vault. Strong SQL, performance tuning, and CI/CD pipeline skills. Experience building and implementing data ingestion and curation processes using big data tools such as Spark (Scala/Python/Java), Databricks, Delta Lake, Hive, HDFS, Sqoop, Kafka, Kerberos, Impala, and CDP 7.x. Hands-on expertise in implementing analytical data stores on the Azure platform using ADLS, Azure Data Factory (ADF), Databricks, Cosmos DB, and CDP 7.x. Experience ingesting huge volumes of data from various platforms for analytics and writing high-performance, reliable, and maintainable ETL code. Proficiency and extensive experience with Spark, Scala, Python, and performance tuning. Hive database management and performance tuning, including partitioning and bucketing. Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance. Strong analytical skills related to working with unstructured datasets. Strong experience in building and designing data warehouses and data stores for analytics consumption, covering both real-time and batch use cases. Ability to interact with business analysts and functional analysts to gather requirements and implement ETL solutions. Performance tuning and problem-solving skills. Code versioning experience using Bitbucket/AzDO. Working knowledge of AzDO pipelines would be a big plus. Experience monitoring performance and advising on necessary infrastructure changes. Eagerness to learn new technologies quickly and deploy them to production. Expertise in technical program delivery across cross-functional/LOB teams. Expertise in driving delivery through collaboration in highly complex, matrixed environments. Strong leadership and negotiation skills. Excellent written and verbal communication skills. Ability to interact with senior leadership teams in IT and business. Preferred Skills: Collibra and insurance industry experience. Azure/Databricks certifications. Expertise in Python and experience writing Azure Functions using Python/Node.js. Experience using Event Hubs for data integrations. Hands-on expertise in implementing analytical data stores on the Azure platform using ADLS, Azure Data Factory, Databricks, and Cosmos DB (Mongo/Graph API). Experience ingesting data using Azure Data Factory and developing complex ETL processes using Databricks.

Azure Data Engineer at Artech LLC | Referso Jobs