Large-scale data pipelines jobs

Research Engineer - Data Eval

Calder · Posted May 12, 2026

Open

Redwood City, California, United States

Apply with RefersoAI Apply on careers page

Research Engineer - Data Eval

Calder · Posted May 12, 2026

Open

Redwood City, California, United States

Apply with RefersoAI Apply on careers page

Software Engineer - Traffic Capacity & Performance

Meta · Posted Nov 3, 2026

Open

Menlo Park, California, United States
On-site
Full-time · Senior
7+ years

Apply with RefersoAI Apply on careers page

The role involves building and extending large-scale simulations of global traffic placement, solving capacity planning as constrained optimization, building production forecasting for peak demand, and leading technical direction across multiple teams.

Responsibilities

Build and extend large-scale simulation of global traffic placement: how traffic shifts across edge sites as demand grows and as sites, peering links and backbone paths fail.
Work with the load-balancing team on simulation fidelity and performance, and validate results against production behavior and live failure tests
Formulate and solve capacity planning as constrained optimization: where to add compute, peering and caches, given space, power, supply, latency and failure constraints.
Build and tune the solvers
Plan capacity for the browsers, proxies and caches Meta's services and AI agents use to reach the internet, and find which one becomes the bottleneck as Meta's internet access needs grow.

Requirements

Bachelor's degree in Computer Science
Bachelor's degree in Computer Engineering
Bachelor's degree in relevant technical field
PhD in operations research
PhD in computer science

Skills

Python C++Java

Director of Engineering - Databricks Commercialization & Marketplace

Databricks · Posted Nov 5, 2026

Open

Bellevue, Washington, United States
Manager
$249K–$350K per year
12+ years

Apply with RefersoAI Apply on careers page

Lead the engineering organization that owns the commercial engine for Databricks, setting product and technology strategy for the money organization.

Compensation

$249,000—$350,000 USD

Responsibilities

Lead engineering organization for commercial engine
Set product and technology strategy
Build leadership bench
Own business outcomes across orgs
Build reusable building blocks for new pricing models

Requirements

Bachelor's in Computer Science

Skills

software engineering distributed systems large-scale data pipelines Kafka Apache Spark Delta Lake cloud (AWS, Azure, GCP)order-to-cash

Software Engineer – Robotics Data Infrastructure

IntelliPro Group Inc. · Posted Nov 5, 2026

Open

San Francisco, California, United States

Apply with RefersoAI Apply on careers page

Software Engineer focused on building and scaling data infrastructure for next-generation robotics and Physical AI, focusing on high-volume data pipelines and scalable production systems.

Responsibilities

Design, build, and maintain scalable data processing and delivery pipelines.
Develop infrastructure for large-scale data transformation, packaging, and storage.
Work with various data formats, schemas, metadata, and custom client requirements.
Build reliable data delivery pipelines using AWS/GCP and distributed processing technologies.
Collaborate directly with customer engineering teams to develop customized solutions.

Requirements

Nice to have: MCAP
Nice to have: ROS
Nice to have: protobuf
Nice to have: Foxglove
Nice to have: multimodal data

Skills

Python Go Rust large-scale data pipelines AWS GCP distributed data processing

Azure Data Engineer

Wire It · Posted Nov 7, 2026

Open

Lisbon, Lisbon District, Portugal +1 more
Hybrid
Contract
No-term full-time contract;
5+ years

Apply with RefersoAI Apply on careers page

Azure Data Engineer

Locations (2)

Lisbon, Lisbon District, Portugal
Porto, Portugal

Skills

Data Warehouse Data Lake Analytics/BI architectures data modelling ETL/ELT large-scale data pipelines Azure Data Factory Azure Data Lake Azure Databricks

Lead Data Engineer

Referrals Only · Posted Nov 5, 2026

Open

Bangalore, India
Hybrid
Senior
10+ years

Apply with RefersoAI Apply on careers page

Lead and manage data engineering projects from inception to completion, designing modern data architecture and overseeing end-to-end data solutions.

Responsibilities

Lead and manage data engineering projects from inception to completion
Collaborate with stakeholders to understand strategic objectives
Design, develop and operate modern data architecture approaches
Create, design and develop intricate data processing pipelines
Collaborate with data scientists to design scalable implementations

Skills

Big Data architecture Data pipelines Distributed systems Data modelling Modern data engineering tools Programming language Large-scale data pipelines Data visualisation Data security

Data Engineer

Zensar · Posted Nov 4, 2026

Open

Pune, Maharashtra, India
Senior
6+ years

Apply with RefersoAI Apply on careers page

Senior Data Engineer responsible for designing, developing, and supporting scalable batch and near real‑time data pipelines, data products, and lakehouse‑ready data assets using Spark, Python, SQL, Hive, Databricks, and cloud‑native technologies.

Responsibilities

Design, develop, and support scalable batch and near real‑time data pipelines, data products, and lakehouse‑ready data assets using Spark, Python, SQL, Hive, Databricks, and cloud‑native technologies.
Build and optimize capabilities supporting the ONE Data Lake and Mastercard's unified data platform strategy.
Develop ingestion, transformation, and orchestration solutions for batch and streaming/near real‑time data patterns using technologies such as Cribl, SSIS, Apache Ozone, Cloudera Data Services, Hive, Spark, Azure, Microsoft Fabric, Databricks, and related ecosystem tools.
Enable trusted, governed, and accessible enterprise data through the implementation of data quality, lineage, and governance standards.
Partner closely with Product, TPM, Data Governance, Architect, and Engineering teams to deliver strategic business outcomes.

Requirements

Nice to have: Cribl or similar observability/log streaming platforms
Nice to have: Microsoft Fabric
Nice to have: OneLake
Nice to have: cloud‑native data platforms
Nice to have: enterprise data governance

Skills

Apache Spark

Large-scale data pipelines jobs

Responsibilities

Requirements

Skills

Responsibilities

Requirements

Skills

Compensation

Responsibilities

Requirements

Skills

Responsibilities

Requirements

Skills

Locations (2)

Skills

Responsibilities

Skills

Responsibilities

Requirements

Skills

Benefits