Back to search

Databricks Lead Engineer

Lead Databricks Lakehouse architecture, platform administration, and workspace management

  • New York, New York, United States
  • Contract
  • $55–$65 per hour
  • Posted Nov 10, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software

Responsibilities

  • Lead Databricks Lakehouse architecture, platform administration, and workspace management
  • Design and implement scalable data solutions using Delta Lake and enterprise data models
  • Develop data pipelines using PySpark, Spark SQL, Auto Loader, and Structured Streaming
  • Manage data governance, access controls, and security using Unity Catalog
  • Integrate Databricks with AWS services, particularly AWS S3
  • Implement CI/CD, workflow orchestration, and deployment processes
  • Leverage MLflow for machine learning lifecycle management
  • Ensure data quality, security, and compliance across enterprise data platforms
  • Coordinate with engineering teams and business stakeholders to drive successful delivery
  • Provide technical leadership and support for production data environments

Requirements

  • Experience in a Insurance

Nice to have: Insurance domain experience

Skills

Databricks LakehousePySparkSpark SQLDelta LakeUnity CatalogMLflowAWSAWS S3Auto LoaderStructured StreamingData ModelingWorkspace AdministrationEnterprise Data ArchitectureData GovernanceCI/CDWorkflow OrchestrationPerformance TuningMetadata ManagementProduction Data PipelinesDatabricksPySparkSpark SQLDelta LakeUnity CatalogMLflowAWSAWS S3Auto LoaderStructured Streaming

More jobs nearby

Full description

Job Role: Databricks Lead Engineer Job Location: New York, NY Job Type: Contract Pay Range: $55 to $65/Hr on C2C Job Overview We are seeking an experienced Databricks Lakehouse Lead / Architect to lead platform architecture, workspace administration, data engineering, governance, and performance Eligible to workimization. The role involves designing scalable data solutions, managing secure data pipelines, and coordinating delivery across enterprise data initiatives. Key Responsibilities Lead Databricks Lakehouse architecture, platform administration, and workspace management. Design and implement scalable data solutions using Delta Lake and enterprise data models. Develop and Eligible to workimize data pipelines using PySpark, Spark SQL, Auto Loader, and Structured Streaming. Manage data governance, access controls, and security using Unity Catalog. Integrate Databricks with AWS services, particularly AWS S3. Implement CI/CD, workflow orchestration, and deployment processes. Eligible to workimize data pipelines and platform performance for reliability and scalability. Leverage MLflow for machine learning lifecycle management. Ensure data quality, security, and compliance across enterprise data platforms. Coordinate with engineering teams and business stakeholders to drive successful delivery. Provide technical leadership and support for production data environments. Qualifications 10+ years of experience in Data Engineering or Data Architecture. Strong expertise in Databricks, PySpark, Spark SQL, and Delta Lake. Hands-on experience with Unity Catalog, AWS, and data governance. Experience in CI/CD, pipeline Eligible to workimization, and production support. Strong leadership and stakeholder management skills. Insurance domain experience preferred. Key Skills Databricks Lakehouse, PySpark, Spark SQL, Delta Lake, Unity Catalog, MLflow, AWS, AWS S3, Auto Loader, Structured Streaming, Data Modeling, Workspace Administration, Enterprise Data Architecture, Data Governance, CI/CD, Workflow Orchestration, Performance Tuning, Metadata Management, Production Data Pipelines, and Insurance Data Domains.

Databricks Lead Engineer at Diverse Lynx | Referso Jobs