Azure Databricks Engineer - Assistant Manager
Design, develop, and support scalable data engineering solutions on Microsoft Azure using Azure Databricks, Apache Spark, Scala, Snowflake, and related technologies.
- Bangalore, Karnataka, India
- Posted Oct 28, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Design and develop scalable, secure, and maintainable data-engineering solutions on Microsoft Azure
- Build and support high-volume data-processing workloads using Azure Databricks, Apache Spark, and Scala
- Develop reliable ETL/ELT pipelines for data ingestion, transformation, processing, and delivery
- Design and implement Snowflake data-processing and data-warehouse solutions
- Develop and optimize SQL queries, stored procedures, functions, and transformation logic
- Integrate Databricks with Snowflake and other enterprise data sources
- Create reusable, modular, and maintainable Scala and Spark components
- Develop batch and streaming data pipelines using Lakeflow Connect, Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Structured Streaming
- Apply Unity Catalog capabilities for data governance, access control, lineage, catalog and schema management, and secure data sharing
- Design and optimize Delta Lake solutions using ACID transactions, schema enforcement and evolution, time travel, change data feed, and table-maintenance practices
- Build incremental and event-driven processing solutions using streaming tables, materialized views, Azure Event Hubs, or Kafka
- Develop and tune Databricks SQL workloads using SQL warehouses, query profiling, Photon acceleration, and cost-optimization practices
- Configure serverless computing, cluster policies, Databricks Runtime versions, workload isolation, and autoscaling based on workload requirements
- Automate Databricks deployments using Declarative Automation Bundles, the Databricks CLI, REST APIs, service principals, and OAuth-based authentication
- Monitor pipeline health, data quality, lineage, streaming backlogs, and resource usage using event logs, system tables, query history, and operational alerts
- Implement data-quality controls, manage schema drift and quarantined records, and build resilient pipelines with checkpoint recovery, idempotent processing, and restart capabilities
- Apply enterprise security practices, including managed identities, private networking, secure secret management, and Azure Key Vault integration
- Support ML flow 3 capabilities for experiment tracking, model evaluation, observability, deployment, and model lifecycle management in Unity Catalog, where applicable
- Tune and optimize Spark/Databricks, SQL, and Snowflake workloads
- Troubleshoot applications, pipeline, deployment, and data-processing issues
- Identify technical risks and production issues and drive them through resolution
- Implement source-control, build, release, and deployment processes using Azure DevOps
- Develop and maintain automated CI/CD pipelines for data-engineering applications
- Support automated deployments across development, test, and production environments
- Participate in application and data-workload migrations to Azure and other modern cloud platforms
- Collaborate with engineering, infrastructure, and operations teams to deliver solutions successfully
- Contribute to solution design, technical reviews, code reviews, and release activities
- Work within Agile, Scrum, or Kanban delivery models and manage deliverables against committed schedules
Requirements
- Bachelor’s degree in computer science, Engineering, Information Technology, or a related discipline
- Experience in a Financial services technology
Nice to have: Azure Data Factory (ADF), Python, Azure SQL or Azure SQL Managed Instance, Oracle or SQL Server, Jira and Confluence, Domain Experience and Certifications, Financial services technology experience, Azure Developer Associate, Azure Data Engineer, Snowflake certification
Skills
Azure DatabricksApache SparkScalaSnowflakeSQLAzure DevOpsCI/CD pipelinesAzure Data FactoryPythonAzure SQLOracleSQL ServerJiraConfluenceAzure DatabricksApache SparkScalaSnowflakeSQLAzure DevOpsAzure Data FactoryAzure SQLAzure Key VaultLakeflow ConnectLakeflow JobsLakeflow Spark Declarative PipelinesAuto LoaderStructured StreamingDelta LakeAzure Event HubsKafkaDatabricks CLIREST APIsOAuthUnity Catalog
Benefits
- Inclusive development opportunities
- Flexible work-life support
- Paid volunteer days
- Vibrant employee networks
Full description
State StreetCompany details are for members
Sign up for a free job alert to see State Street's profile, website and LinkedIn, and the same for every employer on Referso Jobs.