Back to search

Azure Data Engineer

Lead data engineering initiatives, design and implement ETL pipelines, mentor junior engineers, and drive CI/CD automation in Azure environment.

  • Cary, North Carolina, United States
  • Contract
  • On-site
  • $60–$65 per hour
  • Posted Nov 13, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software

Responsibilities

  • Collect, store, process and analyze large datasets to build and implement ETL processes
  • Lead migration of data pipelines from Synapse to Databricks
  • Architect and implement batch and real-time data pipelines using PySpark, Databricks, and Event Hubs
  • Establish data engineering best practices and mentor junior engineers
  • Drive CI/CD automation with AzDO and Git
  • Ensure data quality, governance, and compliance (Collibra)
  • Collaborate with business/tech teams and leverage insurance domain knowledge

Requirements

  • Experience in a insurance

Nice to have: Collibra, insurance industry experience, Azure/Databricks certifications, Python, Node.js, Event Hub, Azure functions

Skills

Azure DatabricksSynapsePySparkEvent HubsAzure Data FactoryADLSCosmos DBSQLSparkScalaPythonKafkaHiveHDFSSqoopKerberosImpalaCDPBitbucketAzDoCI/CDperformance tuningAzure DatabricksSynapsePySparkEvent HubsAzure Data FactoryADLSCosmos DBBitbucketAzDoSparkScalaPythonKafkaHiveHDFSSqoopKerberosImpalaCDP

More jobs nearby

Full description

Title: Azure Data Engineer Location: Cary, NC -Face to face - Onsite Duration: 6-12+ Months contract Rate $60-65/hr on c2c with VMS charges Note- Face to Face Interview Job Description:- • Collect, store, process and analyze large datasets to build and implement extract, transfer, load (ETL) processes. • Lead migration of data pipelines from Synapse to Databricks. • Architect and implement batch + real-time data pipelines using PySpark, Databricks, and Event Hubs. • Establish data engineering best practices and mentor junior engineers. • Drive CI/CD automation with AzDO and Git. • Ensure data quality, governance, and compliance (Collibra). • Collaborate with business/tech teams; leverage insurance domain knowledge where applicable. • Develop reusable frameworks to reduce the development effort involved thereby ensuring cost savings for the projects. • Develop quality code with thought through performance Eligible to workimizations in place right at the development stage. • Appetite to learn new technologies and be ready to work on new cutting-edge cloud technologies. • Work with team spread across the globe in driving the delivery of projects and recommend development and performance improvements. Essential Business Experience and Technical Skills: • Building and Implementing data ingestion and curation process developed using Big data tools such as Spark (Scala/python/Java), Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc. and CDP 7.x • Ingesting huge volumes data from various platforms for Analytics needs and writing high-performance, reliable, and maintainable ETL code Strong • Strong analytic skills related to working with unstructured datasets • Strong experience in building/designing Data warehouses, data stores for analytics consumption On prem and Cloud (real time as well as batch use cases) • Ability to interact with business analysts and functional analysts in getting the requirements and implementing the ETL solutions. Required: • 7+ years in data engineering, 3+ in a lead/senior role. • Strong expertise in Azure Databricks, Synapse, PySpark. • Proven experience with real-time streaming (Event Hubs, Structured Streaming, Kafka). • Deep knowledge of Azure ecosystem (Data Lake, Data Factory, Key Vault). • Strong SQL, performance tuning, and CI/CD pipeline skills. • Excellent leadership and communication skills • Building and Implementing data ingestion and curation process developed using Big data tools such as Spark(Scala/python/Java), Data bricks, Delta Lake, Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc. • Hands on expertise in: Implementing analytical data stores in Azure platform using ADLS/ADF/Data Bricks and Cosmos DB and CDP 7x • Ingesting huge volumes of data from various platforms for Analytics needs and writing high-performance, reliable and maintainable ETL code Strong • SQL knowledge and data analysis skills for data anomaly detection and data quality assurance. • Proficiency and extensive Experience with Spark & Scala, Python and performance tuning is a MUST • Hive database management and Performance tuning is a MUST. (Partitioning / Bucketing) • Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance. • Strong analytic skills related to working with unstructured datasets • Performance tuning and problem-solving skills is a must. • Code versioning experience using Bitbucket/AzDo. Working knowledge of AzDo pipelines would be a big plus. • Monitoring performance and advising any necessary infrastructure changes. • Strong experience in building designing Data warehouses, data stores for analytics consumption. (real time as well as batch use cases) • Eagerness to learn new technologies on the fly and ship to production • Expert in technical program delivery across cross-functional / LOB teams • Expert in driving delivery through collaboration in highly complex, matrixed environment • Possesses strong leadership and negotiation skills • Excellent communication skills, both written and verbal • Ability to interact with senior leadership teams in IT and business Preferred Preferred: • Collibra, insurance industry experience, Azure/Databricks certifications. • Expertise in Python and experience writing Azure functions using Python/Node.js • Experience using Event Hub for data integrations. • Hands on expertise in: Implementing analytical data stores in Azure platform using ADLS/Azure data factory /Data Bricks and Cosmos DB (mongo/graph API) • Experience ingesting using Azure data factory, Complex ETL using Data Bricks. • Eagerness to learn new technologies on the fly and ship to production.

Azure Data Engineer at Diverse Lynx | Referso Jobs