Data Engineer - Data Lake
Develop and deliver scalable transformation frameworks across Hadoop platforms by building robust data pipelines, ingestion frameworks, and automation capabilities.
- Singapore, Singapore, Singapore
- Posted Nov 10, 2026
INDUSTRYSoftware & InternetData & Analytics
Responsibilities
- Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
- Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
- Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
- Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
- Build and maintain batch and near real-time data integration solutions.
- Implement source-to-target mappings and ingestion specifications.
- Develop and support solutions within the Cloudera Data Platform ecosystem.
- Build and maintain data pipelines leveraging Hadoop HDFS, Hive, Impala, Spark, Kafka, YARN, Oozie.
- Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
- Partner with platform teams to ensure optimal cluster utilization and workload management.
- Ensure adherence to Data Lake onboarding standards and governance requirements.
- Investigate and resolve data quality issues and production incidents.
- Ensure compliance with enterprise data governance standards.
- Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps, or equivalent tools.
- Automate deployment, testing, monitoring, and operational support activities.
- Develop reusable frameworks and utilities to improve engineering productivity and platform scalability.
- Provide L3 support for Data Lake application.
- Perform root cause analysis for production incidents and implement permanent fixes.
- Monitor pipeline execution, cluster health, and workload performance.
- Participate in release implementation, deployment validation, and post-production support activities.
Skills
Full description
Company details are for members
Sign up for a free job alert to see Unison Group's profile, website and LinkedIn, and the same for every employer on Referso Jobs.