Databricks Consultant
Design, build and optimise production lakehouse data pipelines on the Databricks platform.
- Cape Town, Western Cape, South Africa
- Posted Nov 11, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Design, build and maintain production data pipelines for ingestion, transformation and delivery of curated data
- Develop and optimise PySpark and Spark SQL transformations, including incremental and batch processing patterns
- Design and operate Delta Lake tables, including table layout, optimisation and file-size management
- Contribute to platform performance and cost efficiency through job, compute and query tuning
- Implement data quality checks, validation, testing and clear technical documentation
- Build and maintain workflow orchestration, job scheduling, retries and alerting for production pipelines
- Apply Unity Catalog for data cataloguing, permissions and lineage where in use
- Follow version control and CI/CD practices for notebooks, packages, jobs and pipeline configuration
- Tune Spark jobs for scale and cost, including partitioning, shuffles, caching and cluster configuration
- Ingest and transform data from enterprise and SaaS sources into well-structured medallion layers
- Troubleshoot failed jobs, slow queries and data defects, and implement lasting technical corrections
- Package reusable pipeline components, parameters and configuration for consistent delivery
- Document pipeline design, table contracts, refresh behaviour and operational steps
Skills
Full description
sabenzait
Company details are for members
Sign up for a free job alert to see sabenzait's profile, website and LinkedIn, and the same for every employer on Referso Jobs.