Senior Data Engineer responsible for building and maintaining large-scale data pipelines, feature engineering, and experimentation infrastructure for a real-time AdTech platform.
Responsibilities
Write and defend diagnostic SQL queries against large-scale production datasets
Build and maintain ingestion pipelines for bid, win, and impression logs into BigQuery
Harmonize fields across independently designed datasets and maintain versioned field mappings
Develop point-in-time-correct feature tables and aggregation pipelines
Design and maintain conversion and labeling pipelines with delayed label handling
Build scalable batch and streaming data pipelines, ingest and quality-check data from various sources into the data warehouse, and orchestrate workflows with Dagster, ensuring data integrity and timely delivery.
Responsibilities
Build scalable batch and streaming pipelines
Process events near real time with deduplication and windowing
Own batch ingestion from databases, files, APIs, and onchain sources
Build incremental dbt models handling late data and duplicates
Orchestrate pipelines in Dagster with sensors, schedules, and alerts
Own the platform layer: the data and ML platform, the infrastructure for ingestion and processing in batch and streaming, data governance and security compliance, the warehouse, and the analytics and BI tooling.
Responsibilities
Design and build data platforms at scale for both batch and streaming, choosing storage, compute, and processing based on latency, cost, and reliability needs.
Own the infrastructure for BigQuery, streaming (such as Pub/Sub or Kafka, and Dataflow or Flink), Dagster+ and Lightdash as code, including environments, CI/CD, and deploys.
Own access and governance: IAM, policy tags, PII contracts, masked and no-PII access paths, and audit logs, across tables and streams.
Provide ML platform capabilities: feature pipelines, training data, and serving real-time features to fraud and risk models.
Build platform tooling: templates, CI checks, schema registry and data contracts, catalog, and lineage.
Review code developed by other developers and provide feedback to ensure best practices (e.g., style guidelines, checking code in, accuracy, testability, and efficiency).
Participate in, or lead design reviews with peers and stakeholders to decide amongst available technologies.
Triage product or system issues and debug/track/resolve by analyzing the sources of issues and the impact on hardware, network, or service operations and quality.
Contribute to existing documentation or educational content and adapt content based on product/program updates and user feedback.
Requirements
Bachelor’s degree or equivalent practical experience.
Experienced Domain Architect specialising in Enterprise Data Warehousing (EDW) and Google Cloud Platform (GCP) to provide technical and architectural leadership for secure, scalable and resilient cloud and data solutions.
Responsibilities
Design and implement secure, scalable and resilient GCP-based architectures
Analyse business and technical requirements and translate them into effective cloud-native solutions
Lead cloud migration initiatives covering lift-and-shift, modernisation and greenfield implementations
Collaborate with application development, operations and security teams to deliver high-quality cloud-native applications
Establish cloud governance models and support compliance with security and cost-management policies
Requirements
B.E./B.Tech
BCA/MCA
BSc/MSc in Computer Science or a related discipline
Software Engineer responsible for designing, developing, and maintaining data pipelines, microservices, and cloud-based solutions at Walmart Global Tech.
Responsibilities
Architect and develop application and pipelines using Java Spring Boot and Scala
Implement data transformation logic to cleanse, normalize, and aggregate data
Develop Web services on Java and Scala using Akka HTTP, Play Framework
Develop and manage Kafka messaging solutions
Integrate data from different systems and sources
Requirements
Bachelor’s degree in Computer Science, Software Engineering, or related technical discipline
Senior Data Engineer responsible for technical design, development, and operational excellence of scalable data pipelines and modern data platform infrastructure within Brokerage Engineering.
Compensation
Base salary range: $120,000 - $143,000
Responsibilities
Lead technical design, development, and optimization of scalable batch and real-time streaming data pipelines
Architect, implement, and maintain modular, version-controlled data models and transformation workflows using SQL and dbt
Build, tune, and scale high-throughput big data processing workloads using Google Cloud Dataproc and Apache Spark on Kubernetes
Provision, configure, and manage resilient cloud data infrastructure using Terraform
Analyze and resolve complex technical problems and production data anomalies
Requirements
Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or equivalent practical experience
Nice to have: Financial services, brokerage technology, trading systems, capital markets, wealth management data domains
Nice to have: Event-driven or real-time streaming architectures (Apache Kafka, Google Cloud Pub/Sub)
Nice to have: Google Cloud Certified Professional Data Engineer
Data EngineerNewRain · San Juan, US
Research Engineer, Gemini Omni Data Infrastructure, DeepMindNewGoogle · London, United Kingdom
Domain Architect (EDW & GCP) -Deputy General ManagerNewVodafone · Pune, India
Senior Data EngineerQuestrade Financial Group · North York, Canada