Data Platform Manager
Responsible for designing, building, and optimizing enterprise wide data platforms within the Data Warehouse.
- Pittsburgh, Pennsylvania, United States
- Full-time
- Hybrid
- Posted Nov 11, 2024
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareData & Analytics
Responsibilities
- Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards
- Support troubleshooting and incident management for data-related issues in production
- Collaborate with business stakeholders, data scientists, and other team members to gather requirements and translate them into technical specifications
- Lead the design, development and deployment of scalable and high-performance data pipelines using Azure Databricks
- Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights
- Define best practices for data ingestion, processing, storage, and governance
- Collaborate with data scientists and analysts to operationalize and deploy machine learning models
- Define the end-to-end Lakehouse architecture using Delta Lake, implementing medallion architecture (Bronze, Silver, Gold layers)
- Oversee the development of robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL
- Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment
- Integrate real-time and batch data sources using Apache Kafka and ADF
- Support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory (ADF)
- Implement Unity Catalog for unified governance, data security, fine-grained access control (RBAC), privacy measures, and data lineage tracking
- Tune Spark jobs and Databricks clusters to maximize throughput while maintaining cost efficiency through auto-scaling and cluster policies
- Expertise in indexing strategies, query optimization, execution plans, and partitioning/sharding
- Orchestrate workflows by integrating Databricks with other Azure services like Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Azure DevOps for CI/CD pipelines
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field
- Experience in a Azure
Nice to have: Microsoft Certified: Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, Azure Solutions Architect Expert, Relational databases, NoSQL databases, Apache Hadoop, Spark, Presto/Trino, Parquet, Avro, ORC, PowerShell, Azure Networking (VNETs), Key Vault, Identity Management, MLflow, Airflow
Skills
PySparkSQLScalaAzure DatabricksAzure cloud technologiesAzure Data FactoryApache KafkaDelta LakePythonAzure DatabricksAzure Data FactoryDelta LakePySparkScalaSQLApache KafkaApache SparkAzure Data Lake Storage (ADLS Gen2)Azure DevOpsUnity CatalogMLflowAirflow
More jobs nearby
Full description
A.C. CoyCompany details are for members
Sign up for a free job alert to see A.C. Coy's profile, website and LinkedIn, and the same for every employer on Referso Jobs.