The Data Engineer builds and operates the pipelines that move data from source systems into the lakehouse and on to analytics and AI consumers, ensuring that the pipelines are automated, performant, secure and governed.
Responsibilities
Build batch, incremental, CDC and streaming ingestion pipelines from source systems into the Bronze layer, using Azure Data Factory and Fabric pipelines
Develop Silver- and Gold-layer transformations in SQL, PySpark and Spark notebooks, including cleansing, conformance, enrichment and business-rule logic
Implement dimensional models, SCDs, surrogate keys and data marts according to the architect's designs
Integrate DQ checks, validation, quarantine and reconciliation into pipelines in partnership with the Senior Data Quality Engineer
Ensure that pipelines publish metadata and lineage to Microsoft Purview
The Senior Data Quality Engineer is accountable for making data trust measurable by owning the enterprise Data Quality (DQ) framework and embedding automated controls at every layer of the lake house.
Responsibilities
Define, operationalize and maintain the enterprise DQ framework, including standards, dimensions, KPIs, thresholds and controls, aligned to ISO 8000 and DAMA-DMBOK2 and to the Dubai Data Law data quality expectations.
Conduct source-data profiling and DQ assessments during Mobilization & Discovery to establish the quality baseline that all improvement KPIs are measured against.
Design and implement DQ checks at the Bronze (ingestion), Silver (transformation/conformance), Gold (curated) and analytical layers.
Build validation, rejection/quarantine, exception-handling and remediation rules directly into data pipelines so that poor-quality data never reaches reporting consumers unflagged.
Establish source-to-target reconciliation controls that evidence completeness, accuracy and consistency for every critical data flow.
The Senior Data Quality Engineer is accountable for making data trust measurable by owning the enterprise Data Quality (DQ) framework and embedding automated controls at every layer of the lake house, from ingestion through to curated and analytical data.
Locations (2)
Abu Dhabi, Abu Dhabi, United Arab Emirates
Delhi, Delhi, India
Responsibilities
Define, operationalize and maintain the enterprise DQ framework, including standards, dimensions, KPIs, thresholds and controls, aligned to ISO 8000 and DAMA-DMBOK2 and to the Dubai Data Law data quality expectations.
Conduct source-data profiling and DQ assessments during Mobilization & Discovery to establish the quality baseline that all improvement KPIs are measured against.
Design and implement DQ checks at the Bronze (ingestion), Silver (transformation/conformance), Gold (curated) and analytical layers.
Build validation, rejection/quarantine, exception-handling and remediation rules directly into data pipelines so that poor-quality data never reaches reporting consumers unflagged.
The Senior Data Quality Engineer is accountable for making data trust measurable by owning the enterprise Data Quality (DQ) framework and embedding automated controls at every layer of the lake house, from ingestion through to curated and analytical data.
Responsibilities
Define, operationalize and maintain the enterprise DQ framework, including standards, dimensions, KPIs, thresholds and controls, aligned to ISO 8000 and DAMA-DMBOK2 and to the Dubai Data Law data quality expectations.
Conduct source-data profiling and DQ assessments during Mobilization & Discovery to establish the quality baseline that all improvement KPIs are measured against.
Design and implement DQ checks at the Bronze (ingestion), Silver (transformation/conformance), Gold (curated) and analytical layers.
Build validation, rejection/quarantine, exception-handling and remediation rules directly into data pipelines so that poor-quality data never reaches reporting consumers unflagged.
Establish source-to-target reconciliation controls that evidence completeness, accuracy and consistency for every critical data flow.
The Data Engineer builds and operates the pipelines that move data from source systems into the lakehouse and on to analytics and AI consumers, ensuring that the pipelines are automated, performant, secure and governed.
Responsibilities
Build batch, incremental, CDC and streaming ingestion pipelines from source systems into the Bronze layer, using Azure Data Factory and Fabric pipelines
Develop Silver- and Gold-layer transformations in SQL, PySpark and Spark notebooks, including cleansing, conformance, enrichment and business-rule logic
Implement dimensional models, SCDs, surrogate keys and data marts according to the architect's designs
Integrate DQ checks, validation, quarantine and reconciliation into pipelines in partnership with the Senior Data Quality Engineer
Ensure that pipelines publish metadata and lineage to Microsoft Purview