Databricks Lead Engineer
Lead Databricks Lakehouse architecture, platform administration, and workspace management
- New York, New York, United States
- Contract
- $55–$65 per hour
- Posted Nov 10, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Lead Databricks Lakehouse architecture, platform administration, and workspace management
- Design and implement scalable data solutions using Delta Lake and enterprise data models
- Develop data pipelines using PySpark, Spark SQL, Auto Loader, and Structured Streaming
- Manage data governance, access controls, and security using Unity Catalog
- Integrate Databricks with AWS services, particularly AWS S3
- Implement CI/CD, workflow orchestration, and deployment processes
- Leverage MLflow for machine learning lifecycle management
- Ensure data quality, security, and compliance across enterprise data platforms
- Coordinate with engineering teams and business stakeholders to drive successful delivery
- Provide technical leadership and support for production data environments
Requirements
- Experience in a Insurance
Nice to have: Insurance domain experience
Skills
Databricks LakehousePySparkSpark SQLDelta LakeUnity CatalogMLflowAWSAWS S3Auto LoaderStructured StreamingData ModelingWorkspace AdministrationEnterprise Data ArchitectureData GovernanceCI/CDWorkflow OrchestrationPerformance TuningMetadata ManagementProduction Data PipelinesDatabricksPySparkSpark SQLDelta LakeUnity CatalogMLflowAWSAWS S3Auto LoaderStructured Streaming
More jobs nearby
Full description
Diverse Lynx
Company details are for members
Sign up for a free job alert to see Diverse Lynx's profile, website and LinkedIn, and the same for every employer on Referso Jobs.