- Remote
- Remote
- Full-time · Mid
- 4+ years
Design, build, and operate scalable lakehouse solutions that ingest, condition, curate, and transform multi-source data into reliable, reusable enterprise data products using Databricks, Apache Spark, Delta Lake, Unity Catalog, structured streaming, and medallion architecture patterns.
Responsibilities
- Design, develop, and maintain scalable batch and streaming data pipelines using Databricks, Apache Spark, PySpark, and Spark SQL.
- Build and manage Delta Lake tables using the medallion (bronze/silver/gold) architecture to deliver reliable, analytics-ready data.
- Develop real-time and near-real-time data ingestion solutions using Spark Structured Streaming and messaging platforms such as Kafka.
- Configure and manage Databricks clusters, jobs, and workflows in production environments.
- Implement data governance, access controls, and security best practices using Unity Catalog.
Requirements
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
- Nice to have: Databricks certification
- Nice to have: MLflow
- Nice to have: Machine learning
- Nice to have: AI