- Bengaluru, Karnataka, India
- Manager
- 9–11 years
Lead Databricks platform administration, security, and architecture across Azure, AWS, and GCP for enterprise client engagements.
Responsibilities
- Oversee Databricks platform administration, security, maintenance, migrations, and integration of new NPI tools across Azure and AWS.
- Design, build, and optimize high-performance ETL/ELT pipelines using Apache Spark, PySpark, SQL, and Delta Lake.
- Implement Medallion Architecture using Bronze, Silver, and Gold data layers.
- Prepare and manage datasets for generative AI and machine learning use cases using MLflow and Vector Search.
- Tune Spark workloads, optimize serverless compute, and use the Photon engine to improve cost and processing efficiency.
Requirements
- Bachelor of Engineering
- Master of Business Administration
- Nice to have: Generative AI dataset preparation
- Nice to have: Multi-cloud Databricks environments
- Nice to have: Tool integration