This is the deep build role. The Senior PySpark and Delta Data Engineer writes the production pipelines that move and reshape data at scale, and is expected to know why a job behaves the way it does rather than just that it finished. The role combines strong software engineering discipline with real Spark depth and hands on ownership of Delta Lake table design and maintenance.
Responsibilities
Build and maintain production PySpark pipelines for batch and streaming workloads
Design Delta Lake tables including partitioning, liquid clustering, Z ordering, and file sizing strategy
Implement incremental processing using Structured Streaming, Auto Loader, and change data feed
Apply merge, upsert, and deduplication logic that holds up under late arriving and out of order data
Tune jobs for runtime and cost by reading the Spark UI and query plans, not by guesswork
Requirements
Nice to have: Databricks Certified Data Engineer Professional
As a Senior Associate in the Data Engineering team, you will play a key role in designing, building, and optimizing modern data platforms and pipelines on Azure and Databricks. You will work within cross-functional teams to deliver scalable, secure, and high-performing data solutions that enable advanced analytics, AI, and business insights for enterprise clients.
Responsibilities
Design, develop, and maintain end-to-end data pipelines across structured, semi-structured, and unstructured data sources
Implement data ingestion, transformation, and orchestration frameworks leveraging Azure Data Factory, Synapse, and/or Microsoft Fabric Data Pipelines
Develop and optimize ETL/ELT processes using Databricks (PySpark, SQL, Delta Lake) to ensure high performance and scalability
Implement and enforce data quality, lineage, and governance practices
Work closely with solution architects to design modern data architectures and ensure compliance with security and privacy standards
Provide production and operational support for enterprise data platforms and engineering workloads, with a primary focus on Databricks, Python, PySpark, and ETL solutions.
Responsibilities
Provide day-to-day production and operational support for Databricks workspaces, jobs, notebooks, workflows, Delta Live Tables or Lakeflow pipelines, clusters, SQL warehouses, libraries, repositories, and related integrations.
Monitor scheduled and streaming workloads, identify failed, delayed, or degraded jobs, assess business impact, restore services, and coordinate recovery within agreed service levels.
Investigate cluster start-up failures, driver and executor errors, autoscaling issues, pool and policy constraints, runtime incompatibilities, library installation failures, and workspace connectivity problems.
Support ETL and ELT pipelines across ingestion, transformation, validation, orchestration, and publishing layers, including batch and streaming processing.
Requirements
Bachelor’s degree in computer science, data engineering, information systems, engineering, or a related discipline, or equivalent professional experience
Design and implement the end-to-end data platform for APEX, integrating call center telephony and quality data from the Genesys A3S Snowflake database and Verint Workforce Management (WFM) feeds.
Responsibilities
Design and implement end-to-end Bronze/Silver/Gold data architecture for Genesys A3S Snowflake and Verint data sources, including data ingestion, transformation, validation, and consumption.
Develop dimensional data models using Kimball principles, including star schemas, fact and dimension tables, grain definition, conformed dimensions, and SCD Type I/II, supporting sales, contact center telephony, agent activity, quality evaluations, and operational reporting.
Build and optimize scalable data pipelines using Databricks, Delta Lake, Python, PySpark, Spark SQL, and T-SQL, applying incremental loading, watermarks, checkpoints, ACID transactions, partitioning, Z-ordering, and query optimization.
Integrate Snowflake sources, vendor flat files, SFTP transfers, and REST APIs, implementing secure connectivity, pagination, rate-limit handling, asynchronous processing, schema evolution, data normalization, and source-to-target reconciliation.
Establish data quality, integrity, and audit controls, including deduplication, referential integrity, row counts, checksum/hash comparisons, late-arriving data handling, and detection and remediation of missing records, data drift, and schema changes.
Lead a team responsible for data and full-stack engineering, partnering closely with Product Management to translate product vision into high-quality delivered experiences.
Compensation
The anticipated base pay compensation range for this position is $124,200.00 – $207,000.00. This role is eligible for an incentive target of up to 15% or $, based on the achievement of individual and company performance objectives in accordance with the current terms of the incentive program which are subject to change.
Responsibilities
Lead a team of software and data engineers, fostering a collaborative, engaging environment and a culture of engineering excellence.
Provide hands-on technical guidance through code reviews, pair programming, and day-to-day coaching.
Partner with Product Management, business stakeholders, and technical leadership to define the roadmap, align on architecture, prioritize features, and guide delivery.
Foster a culture of continuous improvement by identifying opportunities to strengthen development practices, improve efficiency, and deliver high-quality software.
Plan and allocate team capacity based on skill sets, project needs, and delivery timelines to achieve commitments effectively.
Requirements
Nice to have: Partnering with Product Management and UX to deliver data-intensive products
Databricks MLOps Engineer with 6+ years in ML engineering, MLOps or data engineering, including 3+ years building ML pipelines on Databricks, to take models from notebooks to tested, monitored production.
Responsibilities
Build feature pipelines on Delta tables and manage features in Unity Catalog feature tables.
Automate training and retraining with Databricks Workflows, including parameterized and scheduled runs.
Track experiments and register, version and promote models with MLflow and the Unity Catalog model registry, using aliases for champion and challenger models.
Deploy models to Databricks Model Serving (real‑time) and batch inference jobs; run A/B and shadow deployments.
Monitor models with Lakehouse Monitoring and inference tables: data drift, prediction drift, latency, error rates and alert thresholds.
Requirements
Bachelor's or Master's degree in Computer Science, Data Science, Engineering or a related field.
Nice to have: Databricks Certified Machine Learning Associate or Professional
Design, develop, and support enterprise scale data and analytics solutions using the Databricks Lakehouse Platform, supporting client engagements across the United States.
Compensation
A reasonable estimate of the current range for this role in the U.S. is $58,100.00 $128,500.00.
Responsibilities
Design, develop, and support enterprise scale data and analytics solutions using the Databricks Lakehouse Platform
Collaborate with Enterprise Technology Practice leaders, solution architects, and client teams to deliver POCs, pilots, and production implementations
Build and maintain scalable data pipelines using Databricks, Delta Lake, Apache Spark, Unity Catalog, and Databricks Workflows
Develop ETL/ELT processes for batch and streaming data integration
Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency
Requirements
Bachelor's degree in computer science on related field
Nice to have: Databricks Data Engineer Associate/Professional or Databricks Solutions Architect certification