Automation of internal processes in MDSS2, using the strategic Big Data infrastructure (HDFS, Apache Spark (PySpark)) and Python, including but not limited: Market data daily monitoring process, RFI monitoring process, etc.
Responsibilities
Automation of internal processes in MDSS2 using Big Data infrastructure (HDFS, Apache Spark (PySpark)) and Python
Understanding quantitative methodologies and business requirements
Designing and implementing solutions that fit into the overall strategic framework
Conducting testing to ensure the appropriateness of the implementations
Maintaining the implementation to incorporate user feedback as well as performance enhancements
Requirements
Nice to have: financial market data across different asset classes
The Enterprise Data Platform team is looking for a Principal Data Engineer to help revolutionize how we think about and deliver customer insights, playing an important role in the future success of the company.
Locations (2)
United States
Ontario, Canada
Compensation
$137,800—$172,250 USD
Responsibilities
Architecting large-scale data intelligence solutions, performing architectural assessments, crafting architectural options and analysis around Snowflake Data Warehouse and/or Google BigQuery
Design and evolve an open lakehouse on AWS: S3 with Apache Iceberg/Delta table formats, a medallion architecture (bronze → silver → gold), and Trino as the federated query engine
Finalize preferred solution alternative working with IT and Business stakeholders to build data ingestion pipeline using tools like AWS EMR, Snowpipe, AWS DMS, etc.etc
Developing ETL pipelines in and out of data the warehouse using but not limited to Python, Scala,SnowSQL, etc.
Lead and build a robust data platform for Instacart Ads, designing systems that enforce data compliance and enable self‑service for internal consumers.
Senior Ads Data Engineer responsible for designing, building, and maintaining scalable data infrastructure on Databricks to support Life360’s advertising and analytics needs.
Locations (2)
United States, United States
Canada, Canada
Compensation
For candidates based in the US, the salary range for this position is $129,000 to $192,000 USD. For candidates based out of Canada, the salary range for this position is $149,500 to $177,500 CAD.
Responsibilities
Enhance, extend, and operate existing batch and streaming pipelines and Delta Lake tables on Databricks
Partner with business analysts, ad strategists, and sales team to translate questions into data products
Build and maintain dbt models and tests for curated datasets
Manage Databricks and AWS infrastructure as code with Terraform
Integrate and reconcile data from ad platforms, CRM, partner feeds, and internal systems
Requirements
Bachelor’s degree or equivalent experience in Computer Science, Engineering, or a related field
Design, develop, and maintain robust data integration solutions, focusing heavily on real-time data streaming via Apache Kafka and cloud-native ETL pipeline orchestration using AWS Glue.
Responsibilities
Build and optimize scalable streaming solutions using Apache Kafka, Kafka Connect, and Kafka Streams
Architect and manage ETL/ELT pipelines and the Data Catalog using AWS Glue
Connect cloud applications and third-party services using AWS Lambda, S3, API Gateway, SQS, SNS, and EventBridge
Champion event-driven architecture and microservices principles
Mentor junior engineers and provide technical leadership
Own orchestration patterns in Dagster: asset-based pipelines, code-location design, cross-asset lineage, sensors and schedules, partitioning and backfills
Requirements
Bachelor or Master’s degree in Computer Science, Information Technology or Equivalent
Nice to have: semantic layers (Cube, dbt Semantic Layer)