Senior Data Engineer
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Coimbatore, Tamil Nadu, India
- Posted Nov 10, 2026
INDUSTRYSoftware & InternetData & Analytics
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Work on Databricks infrastructure, including workspace configuration, clusters, compute resources, access controls, and performance optimization.
- Apply Data Governance principles, including data quality, data lineage, metadata management, data cataloging, and access controls.
- Work with Unity Catalog to manage data permissions, governance policies, and data assets.
- Develop and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, and Gold).
- Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
- Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
- Implement data validation, error handling, logging, monitoring, and data quality checks.
- Optimize Spark workloads, SQL queries, and data processing pipelines for performance and scalability.
- Support data discovery, metadata documentation, and governance standards across data platforms.
- Collaborate with cross-functional teams to ensure data accuracy, consistency, security, and compliance.
- Troubleshoot data pipeline and infrastructure issues and recommend improvements to platform reliability.
- Contribute to enterprise data management and governance best practices.
Skills
DatabricksPySparkPythonSQLDelta LakeUnity CatalogData GovernanceData QualityMetadata ManagementData LineageData CatalogingData SecuritySpark Performance TuningGitCI/CDProduction DeploymentDatabricks Lakehouse PlatformDelta LakeUnity CatalogDatabricks JobsDatabricks WorkflowsDatabricks ClustersDatabricks Notebooks
Benefits
- Competitive salary
- Strong insurance package
- Extensive learning and development resources
Full description
ShyftLabsCompany details are for members
Sign up for a free job alert to see ShyftLabs's profile, website and LinkedIn, and the same for every employer on Referso Jobs.