Lead Data Engineer – AI & Automation
Drive the adoption of AI-assisted engineering and intelligent automation across the enterprise data platform, focusing on defining and executing an AI-first Data Engineering strategy.
- Raleigh, North Carolina, United States
- Contract
- Hybrid
- Posted Nov 10, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareData & Analytics
Responsibilities
- Define and execute the AI-enabled Data Engineering strategy to improve engineering velocity, quality, reliability, scalability and operational efficiency.
- Identify, assess and prioritize high-value AI and automation use cases across Data Engineering, Data Warehouse and BI teams.
- Develop reusable patterns and accelerators for introducing AI into existing Data Engineering and platform operations.
- Build an AI-first engineering culture by adopting driving, enablement, governance and measurable engineering outcomes.
- Lead adoption of GitHub Copilot, Snowflake Cortex and Databricks Genie to accelerate engineering activities and improve developer productivity.
- Implement AI-assisted approaches for generating SQL, Python, PySpark, pipeline code, unit tests, documentation and deployment artifacts.
- Automate generation of standardized, reusable data pipelines and engineering components.
- Target automation of approximately 30–50% of repetitive pipeline development activities, where technically and operationally appropriate.
- Build reusable engineering templates, prompts, coding standards and AI-assisted development patterns.
- Implement AI-driven monitoring and anomaly detection to proactively identify Data quality issues, Pipeline failures, Schema changes and drift, Processing anomalies, and Performance degradation.
- Design self-healing data pipelines capable of detecting and automatically resolving known operational issues.
- Apply intelligent automation to incident triage, troubleshooting and root-cause analysis.
- Reduce manual L1/L2 operational effort through automated diagnosis, remediation and preventive monitoring.
- Establish appropriate human-in-the-loop controls for scenarios where automated remediation could create downstream risk.
- Leverage Snowflake AI and Cortex capabilities to improve engineering productivity and platform intelligence.
- Identify expensive or inefficient queries and workloads and implement appropriate optimization strategies.
- Improve warehouse utilization, workload management, query performance and overall platform cost efficiency.
- Utilize Databricks Genie and Databricks AI capabilities to improve data discovery, analytics accessibility and engineering productivity.
- Work with platform teams to establish measurable FinOps outcomes around compute consumption and cloud cost optimization.
- Champion the use of AI across the complete engineering lifecycle, including Requirements analysis and refinement, Solution and pipeline design, Code generation and refactoring, Unit and integration testing, SQL and PySpark optimization, Code reviews, Technical documentation, CI/CD and deployment automation, Production monitoring, Incident analysis, Root-cause analysis, Technical debt identification and remediation.
Requirements
- Experience in a Reinsurance
- Experience in a Bordereaux management and reporting
Nice to have: Azure, AWS, Azure Data Factory, dbt, Airflow, data observability frameworks, MLOps, LLMOps, RAG, AI agent/agentic workflow concepts, DataOps, DevOps, FinOps, Platform Engineering
Skills
AI ArchitectData EngineeringSnowflakeDatabricksSQLPythonPySparkETL/ELT pipelinesAI-assisted engineering toolsGitHub CopilotSnowflake CortexDatabricks GenieCI/CDDevOpsdata observabilityanomaly detectionGenerative AILLMsprompt engineeringenterprise AI governanceGitHub CopilotSnowflake CortexDatabricks GenieSnowflakeDatabricksSQLPythonPySpark
More jobs nearby
Full description
Infosys Pontoon
Company details are for members
Sign up for a free job alert to see Infosys Pontoon's profile, website and LinkedIn, and the same for every employer on Referso Jobs.