Back to search

Senior Data Engineer

Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks, focusing on large-scale distributed data processing.

  • Coimbatore, Tamil Nadu, India
  • Full-time
  • Posted Nov 13, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareData & Analytics

Responsibilities

  • Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks
  • Develop data ingestion and transformation workflows using Python, SQL, and PySpark
  • Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms
  • Integrate data from multiple sources, including relational databases, APIs, and cloud storage
  • Develop and optimize data models, fact tables, dimension tables, and data warehouse structures
  • Perform data cleansing, validation, deduplication, and aggregation
  • Implement and manage workflows using Azure Data Factory and Apache Airflow
  • Optimize SQL queries, Spark jobs, and data pipelines for better performance
  • Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps
  • Troubleshoot production issues, perform root cause analysis, and provide production support

Requirements

  • Experience in a Cloud-based data engineering
  • Experience in a Data warehousing

Nice to have: Pandas, NumPy, OOP concepts, API integration

Skills

PythonSQLPySparkAzure Data FactoryAzure DatabricksApache SparkETL/ELT pipeline developmentChange Data Capture (CDC)Dimensional and relational data modelingGitGitHubAzure DevOpsAzure Data Lake Storage Gen2Azure Blob StorageAzure Synapse AnalyticsAzure Key VaultAzure MonitorApache AirflowMicrosoft SQL ServerPostgreSQLMySQLOracleCSVJSONXMLParquetDelta Lake

Full description

Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks. Develop data ingestion and transformation workflows using Python, SQL, and PySpark. Work with Apache Spark and PySpark for large-scale distributed data processing. Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms. Integrate data from multiple sources, including relational databases, APIs, and cloud storage. Work with Azure Data Lake Storage Gen2 and Azure Blob Storage. Develop and optimize data models, fact tables, dimension tables, and data warehouse structures. Perform data cleansing, validation, deduplication, and aggregation. Implement and manage workflows using Azure Data Factory and Apache Airflow. Optimize SQL queries, Spark jobs, and data pipelines for better performance. Work with Azure Synapse Analytics and relational databases. Implement error handling, logging, monitoring, and alerting for production pipelines. Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps. Troubleshoot production issues, perform root cause analysis, and provide production support. Maintain technical documentation and collaborate with cross-functional teams. Requirements Programming Languages Python, SQL, PySpark. Pandas, NumPy, OOP concepts, exception handling, and API integration. Big Data Technologies Apache Spark and PySpark. Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions. Partitioning, caching, and performance optimization. Azure Cloud Technologies Azure Data Factory (ADF). Azure Databricks. Azure Data Lake Storage Gen2 (ADLS Gen2). Azure Blob Storage. Azure Synapse Analytics. Azure Key Vault and Azure Monitor. ETL & Data Engineering ETL/ELT pipeline development. Data ingestion, transformation, and migration. Change Data Capture (CDC). Incremental loading and full loading. Batch and distributed processing. Database Technologies Microsoft SQL Server, PostgreSQL, MySQL, and Oracle. Complex SQL queries, indexing, stored procedures, and query optimization. Data Modeling & Warehousing Dimensional and relational data modeling. Fact and dimension tables. Data normalization and denormalization. Schema design and data warehouse development. Workflow Orchestration & DevOps Azure Data Factory and Apache Airflow. Git, GitHub, Azure DevOps, and CI/CD pipelines. File Formats CSV, JSON, XML, Parquet, and Delta Lake. 4–5 years of relevant experience in Data Engineering. Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL. Experience developing and optimizing scalable data pipelines. Good understanding of cloud-based data engineering and data warehousing concepts. Strong analytical, debugging, and problem-solving skills. Good communication and collaboration skills.

Senior Data Engineer at infynd | Referso Jobs