Back to search

GCP Senior Data Engineer

Senior Data Engineer responsible for building and maintaining data pipelines using GCP, handling sensitive health data (HIPAA/PHI), and supporting ML workflows.

  • Remote
  • Contract
  • Remote
  • Posted Nov 17, 2026
INDUSTRYHealthcareHome Health & Senior Care

Responsibilities

  • Stream health, benefits, news and click-through files (CSV/Parquet) through Kafka and Dataflow into Cloud Storage and BigQuery
  • Run ETL/ELT jobs on Dataflow and Dataproc Spark to produce clean, normalized BigQuery datasets
  • Build 360° denormalized views that combine prescription (Rx), benefits, behavioral and clinical signals
  • Load the GCP Feature Store, build batch and online feature paths, create train/eval splits and version datasets, and monitor data drift with Vertex AI
  • Bring in behavioral event data from Adobe Analytics and line it up with clinical and benefits data
  • Set up SLAs, alerting, schema validation and anomaly detection for data quality in BigQuery
  • Set recovery time and recovery point targets (RTO/RPO), replicate BigQuery across regions, maintain DR buckets and Feature Store snapshots, and test failover runbooks
  • Set BigQuery table expiry, move older data to cheaper Cloud Storage tiers (Nearline, Coldline, Archive) and meet HIPAA retention rules
  • Apply field-level encryption and masking and de-identify data (Safe Harbor or Expert Determination) before training, enforcing access controls and audit logging for HIPAA/PHI

Requirements

  • Experience in a HIPAA
  • Experience in a PHI

Nice to have: Adobe Experience Platform, Looker, GCP customer-managed encryption keys (CMEK), HIPAA business associate agreement (BAA) awareness, NIST/HITRUST, healthcare data experience (Rx, clinical or benefits)

Skills

BigQueryKafkaDataflow/BeamSparkPub/SubComposer/AirflowSQLdata modeling (lakehouse)Feature StoresVertex AI PipelinesML lifecycleDR designarchival and retention for regulated dataHIPAA/PHI controlsGCPBigQueryKafkaDataflowSpark (DataSpark)Vertex AICloud StorageAdobe AnalyticsPythonJavaNode.js

Full description

Senior Data Engineer Remote Duration : long Term Mode of interview : 2 videos Experience 7–9 years in data engineering or ML data engineering, in production GCP Stack GCP, BigQuery, Kafka, Dataflow, Spark (DataSpark), Vertex AI, Cloud Storage, Adobe Analytics, Python/Java/Node.js Key asks Ingestion: Stream health, benefits, news and click-through files (CSV/Parquet) through Kafka and Dataflow into Cloud Storage and BigQuery. Transformation: Run ETL/ELT jobs on Dataflow and Dataproc Spark to produce clean, normalized BigQuery datasets. Curated marts: Build 360° denormalized views that combine prescription (Rx), benefits, behavioral and clinical signals. Feature Store and ML datasets: Load the GCP Feature Store, build batch and online feature paths, create train/eval splits and version datasets. Monitor data drift and feature freshness with Vertex AI. Adobe Analytics: Bring in behavioral event data and line it up with clinical and benefits data. Data quality and BigQuery operations: Set up SLAs, alerting, schema validation and anomaly detection. Manage partitioning, schema changes and cost. Disaster recovery: Set recovery time and recovery point targets (RTO/RPO), replicate BigQuery across regions, maintain DR buckets and Feature Store snapshots, and test failover runbooks. Archival: Set BigQuery table expiry, move older data to cheaper Cloud Storage tiers (Nearline, Coldline, Archive) and meet HIPAA retention rules. HIPAA and patient health information (PHI): Apply field-level encryption and masking and de-identify data (Safe Harbor or Expert Determination) before training. Enforce access controls and audit logging. Documentation: Write up data contracts, data lineage, Feature Store definitions, DR runbooks and archival policies. Must-have skill Core data skills: BigQuery (advanced), Kafka, Dataflow/Beam, Spark (PySpark or Scala), Pub/Sub, Composer/Airflow, plus strong SQL and data modeling (lakehouse). ML data work: Feature Stores, Vertex AI Pipelines or similar MLOps tools, and the ML lifecycle. Proven experience with: DR design, archival and retention for regulated data, and hands-on HIPAA/PHI controls. Nice to have Adobe Experience Platform, Looker, GCP customer-managed encryption keys (CMEK), HIPAA business associate agreement (BAA) awareness, NIST/HITRUST, and healthcare data experience (Rx, clinical or benefits).

GCP Senior Data Engineer at Echo IT Solutions, Inc | Referso Jobs