Back to search

Technical Integration Lead Databricks

Design, build, and deliver integrations that consume Master Data products published in Databricks and feed them into an enterprise commercial CRM platform as part of a large-scale CRM modernization program.

  • Indianapolis, Indiana, United States
  • Contract
  • On-site
  • $56–$65 per hour
  • Posted Nov 18, 2026
INDUSTRYLife Sciences & ResearchBiotechnology

Responsibilities

  • Personally perform coding, code review, and deployment activities
  • Own the end-to-end integration lifecycle: design specification, build, unit testing, deployment across dev / sandbox / UAT / prod, runbooks, and knowledge transfer
  • Consume Databricks data products (Delta Lake tables via Unity Catalog) as the source of master data
  • Move master data from Databricks into the CRM using appropriate patterns (Apache Iceberg, Spark/PySpark, or API-based delivery)
  • Build transformation and mapping logic (PySpark / Spark SQL) to reconcile data-product schema with target CRM data model
  • Re-platform existing master-data integrations onto Databricks-data-product and modern integration patterns
  • Implement observability, data-quality checks, error handling, data-gap detection, and reprocessing across the Databricks-to-CRM flows
  • Participate in and lead all relevant client meetings
  • Collaborate with the second System Integrator (SI) partner

Requirements

  • Experience in a Life Sciences
  • Experience in a Pharmaceutical

Nice to have: Informatica IICS and ETL (PowerCenter), GxP, CSV, 21 CFR Part 11

Skills

DatabricksDelta LakeUnity CatalogETLPySparkSpark SQLREST API designMaster Data / MDMAWS Data servicesVeeva Vault CRM integrationDatabricksDelta LakeUnity CatalogAWSPySparkSpark SQLVeeva Vault CRM

More jobs nearby

Full description

Job Title: Technical Integration Lead Databricks Data Products Location: Indianapolis, IN Duration: 6+ months Contract Skills: Databricks Hourly Rate: $56-$65/hour Required Skills & Qualifications: Job Description Integration Lead – Databricks Data Products & Enterprise CRM Integration Role Type Technical Integration Lead (Hands-On) Industry Life Sciences / Pharmaceutical Primary Platform Databricks Data Products, ETL, Veeva Vault CRM Client Commercial CRM for Life Sciences Delivery Model Onsite – Offshore Delivery Context Multi-Vendor (Dual System Integrator) Experience 7+ years IT, 5+ years on ETL, 3+ years on Databricks Location Onsite Role Overview We are seeking an experienced, hands-on Integration Lead to design, build, and deliver integrations that consume Master Data products published in Databricks and feed them into an enterprise commercial CRM platform, as part of a large-scale CRM modernization program. Master Data sources (customer, account, and product master) are exposed as governed data products in Databricks – curated Delta tables registered in Unity Catalog. This role owns the consumption of those data products and their reliable, validated delivery into the target CRM and enterprise ecosystem, while re-platforming existing integrations onto modern, cloud-native patterns. This is a working, hands-on leadership role. The person leads the integration workstream and personally performs the coding, code review, and deployment work – not oversight alone. Every skill listed is expected to be backed by direct, hands-on delivery experience. Key Responsibilities 1. Hands-On Integration Delivery • Personally perform coding, code review, and deployment activities, and any other assigned work related to this transformation – leading the team does not replace individual hands-on contribution. • Must have hands-on experience across all required skills; the person is expected to build, not only direct others who build. • Own the end-to-end integration lifecycle: design specification, build, unit testing, deployment across dev / sandbox / UAT / prod, runbooks, and knowledge transfer. • Produce design specifications, deployment steps, and rollback approaches, and deliver them hands-on. 2. Databricks Data-Product Consumption & Integration • Consume Databricks data products (Delta Lake tables via Unity Catalog) as the source of master data, respecting the data-product contract – schema, refresh cadence, quality guarantees, and access model. • Move master data from Databricks into the CRM using the appropriate pattern – Apache Iceberg interoperability, Spark / PySpark-based movement, or API-based delivery – selecting the right mechanism per integration by volume, latency, and directionality. • Build transformation and mapping logic (PySpark / Spark SQL) to reconcile the data-product schema with the target CRM data model, including handling master-data change (deltas, data-change requests, reference data). • Re-platform existing master-data integrations onto the Databricks-data-product and modern integration patterns, ensuring parity, reconciliation, and no data loss during transition. • Implement observability, data-quality checks, error handling, data-gap detection, and reprocessing across the Databricks-to-CRM flows. 3. Client Engagement & Meetings • Participate in and lead all relevant client meetings – design reviews, working sessions, governance forums, and status discussions – representing the integration workstream directly with the client. • Communicate technical decisions, risks, and dependencies clearly to client stakeholders and delivery teams. 4. Multi-Vendor (Dual-SI) Collaboration • Collaborate with the second System Integrator (SI) partner responsible for Consultantly testing and validating this team’s build. • Review the second SI’s test findings, triage them, and resolve the defects – ensuring fast, clean cross-vendor root-cause and closure rather than back-and-forth. • Maintain clean build-to-test handoffs and shared traceability so defects are resolved efficiently across vendors. 5. Patterns, Standards & Data Partnership • Define reusable integration patterns and interface contracts for master-data flows that other build and assurance teams can work to. • Partner with data-product owners and data-engineering teams to align on data-product schemas, versioning, refresh frequency, access, and change management. • Ensure data integrity, security, privacy, and, where applicable, regulated-industry compliance across connected systems. Must-Have Technical Skills All of the following require direct, hands-on experience: • Databricks – hands-on with Delta Lake, Unity Catalog, and consuming / publishing data products; lakehouse / medallion architecture. • AI – hands-on experience applying AI in a data, integration, or software-development context. • Data services on AWS – hands-on experience building data and transformation services / pipelines on AWS. • API – hands-on REST API design, build, and implementation at scale. • PySpark – strong, hands-on PySpark and Spark SQL for data movement and transformation. • Master Data / MDM – hands-on experience integrating customer, account, and product master data into downstream systems, including schema mapping, reconciliation, and master-data change handling. • Demonstrated ability to lead a build team while remaining hands-on – doing the build, review, and deployment work directly. Other Skills • Informatica – IICS and ETL (PowerCenter); ability to understand and re-platform existing Informatica integrations. • Veeva CRM and Veeva Vault CRM integration knowledge – Vault REST API, Direct Data API, Spark Messaging, Vault Loader, VQL, and the Vault data model. • Experience remediating or migrating enterprise CRM platforms. • Regulated-industry (pharma / life sciences) experience; GxP / CSV / 21 CFR Part 11 familiarity. • Experience in a multi-vendor / dual-SI delivery model; global, multi-geography delivery with data-residency awareness. • Databricks certification. Leadership & Delivery Competencies • A true practitioner who personally builds – owns code, reviews, and deployments, not only direction. • Strong client-facing communication; able to lead client meetings and represent the integration workstream credibly. • Collaborative across data-engineering, platform, and the second SI partner – resolves cross-vendor test findings without friction. • Strong analytical thinking, root-cause discipline, and problem-solving under delivery pressure. • Willing and able to be on-site in Indianapolis and travel for key meetings. Role Criticality This role is critical because the Integration Lead is the hands-on owner of the integrations that bridge governed Databricks & other data products into the enterprise CRM. Accuracy in these flows is non-negotiable – an error in master data reaches the field and business operations quickly. The person in this role must combine deep, hands-on Databricks, AWS, and PySpark engineering with the ability to lead client meetings, resolve cross-vendor test findings, and deliver reliably in a regulated, multi-vendor program Role Descriptions: We are seeking an experienced| hands-on Integration Lead to design| build| and deliver integrations that consume Master Data products published in Databricks and feed them into an enterprise commercial CRM platform| as part of a large-scale CRM modernization program. Master Data sources (customer| account| and product master) are exposed as governed data products in Databricks curated Delta tables registered in Unity Catalog. This role owns the consumption of those data products and their reliable| validated delivery into the target CRM and enterprise ecosystem| while re-platforming existing integrations onto modern| cloud-native patterns. This is a working| hands-on leadership role. The person leads the integration workstream and personally performs the coding| code review| and deployment work not oversight alone. Every skill listed is expected to be backed by direct| hands-on delivery experience.

Technical Integration Lead Databricks at Diverse Lynx | Referso Jobs