- Madrid, Madrid, Spain
- Hybrid
Lead data engineering initiatives, migrating Hadoop infrastructure to cloud, building CI/CD pipelines, and ensuring data quality and consistency.
Responsibilities
- Migrate existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service
- Implement data transformation and quality to ensure data consistency and accuracy
- Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations
- Set up CI/CD pipelines to automate deployments, unit testing and development management
- Write and conduct unit and validation tests to ensure accuracy and integrity of code developed
Requirements
- Nice to have: Kubernetes containerization
- Nice to have: Dremio as tool to virtualize data
- Nice to have: Elasticsearch
- Nice to have: Kibana
- Nice to have: Streaming process (Kafka, event steam…)