Back to search

Databricks Data Engineer

  • Mexico City, Distrito Federal, Mexico
  • Posted Nov 13, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software

Responsibilities

  • Design and implement end-to-end data pipelines using Azure Databricks including batch and streaming workflows
  • Build and maintain Delta Lake architecture including bronze, silver, and gold layers
  • Develop scalable ETL/ELT pipelines using Apache Spark (PySpark / Scala)
  • Optimize Databricks jobs for performance, cost, and reliability — including cluster tuning, caching, and query optimization
  • Implement Delta Live Tables (DLT) for declarative pipeline development
  • Build and maintain Unity Catalog for data governance, lineage, and access control
  • Collaborate with data architects to design data models (medallion architecture, star/snowflake schemas)
  • Integrate Databricks with cloud services: Azure Data Factory, Azure Synapse, S3, ADLS Gen2
  • Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps / GitHub Actions
  • Ensure data quality, monitoring, and alerting across data pipelines
  • Contribute to client workshops, technical design sessions, and solution architecture reviews

Requirements

Nice to have: Snowflake, IICS, Stonebranch, YAML

Skills

DatabricksPySparkScalaDelta LakeSQLPythonMicrosoft AzureApache AirflowAzure Data FactoryDatabricks WorkflowsGitCI/CDAgile/ScrumAzure DatabricksDelta LakeApache SparkAzure DevOpsGitHub ActionsAzure Data FactoryAzure SynapseS3ADLS Gen2Unity CatalogAzure Data FactoryApache Airflow

Full description

Databricks Data Engineer Key Responsibilities Design and implement end-to-end data pipelines using Azure Databricks including batch and streaming workflows Build and maintain Delta Lake architecture including bronze, silver, and gold layers Develop scalable ETL/ELT pipelines using Apache Spark (PySpark / Scala) Optimize Databricks jobs for performance, cost, and reliability — including cluster tuning, caching, and query optimization Implement Delta Live Tables (DLT) for declarative pipeline development Build and maintain Unity Catalog for data governance, lineage, and access control Collaborate with data architects to design data models (medallion architecture, star/snowflake schemas) Integrate Databricks with cloud services: Azure Data Factory, Azure Synapse, S3, ADLS Gen2 Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps / GitHub Actions Ensure data quality, monitoring, and alerting across data pipelines Contribute to client workshops, technical design sessions, and solution architecture reviews Nice to have if knowledge in Snowflake, IICS, Stonebranch Should be good with YAML files work - metadata/config driven pipelines for platform requires handling data ingestion and moving data from Bronze to Silver using configuration files (YAML), Required Skills & Experience Core Technical Skills 3–8 years of experience in data engineering, data warehousing, or big data solutions Hands-on expertise with Databricks (Databricks Certified Associate/Professional preferred) Strong proficiency in PySpark and/or Scala for large-scale data processing Experience with Delta Lake — ACID transactions, schema evolution, time travel, merge operations Proficiency in SQL (SparkSQL, HiveQL) and Python Experience with cloud platforms: Microsoft Azure Experience with orchestration tools: Apache Airflow, Azure Data Factory, Databricks Workflows Data Architecture Strong understanding of medallion architecture (Bronze / Silver / Gold) Experience designing data lakehouses, data warehouses, or data mesh architectures Familiarity with data modeling concepts — dimensional modeling, data vault Knowledge of data governance principles and cataloging tools (Unity Catalog, Purview, Collibra) DevOps & Collaboration Experience with Git and version control best practices Familiarity with CI/CD tools (Azure DevOps, GitHub Actions, Jenkins) Experience working in Agile/Scrum delivery environments Strong communication skills — ability to explain technical concepts to non-technical stakeholders

Databricks Data Engineer at SGF Global Inc | Referso Jobs