- Madrid, Madrid, Spain
- Hybrid
Data Engineer responsible for migrating Hadoop infrastructure to cloud, implementing data pipelines, and ensuring data quality.
Responsibilities
- Migrate existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service
- Implement data transformation and quality to ensure data consistency and accuracy
- Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations
- Set up CI/CD pipelines to automate deployments, unit testing and development management
- Write and conduct unit and validation tests to ensure accuracy and integrity of code developed