- Remote, United States
- Remote
- Part-time · Lead
- 8+ years
The Senior Lead Data Engineer on the Prism team sets the technical direction for designs and delivers the core data engineering infrastructure for Lighthouse's eDiscovery AI products — including AI-enabled TAS (Technology Assisted Solutions) and Generative AI document analysis services.
Compensation
This role will be eligible to participate in an annual bonus or incentive program.
Responsibilities
- Set the technical direction and own the end-to-end architecture for data engineering across Prism eDiscovery AI products, ensuring designs are scalable, reliable, secure, and cost-effective.
- Lead the design, build, and management of data preprocessing, validation, and transformation pipelines that extract metadata and full text from heterogeneous ESI corpora at multi-million document scale.
- Lead the build and operation of Data Orchestration platforms that coordinate training data preparation, model scoring, embedding generation, and result delivery across eDiscovery AI workflows.
- Architect and implement scalable, performant data pipelines that cleanse, structure, and integrate disparate ESI datasets into optimal formats for encoder model training (BERT fine-tuning) and LLM context preparation.
- Direct the design and evolution of infrastructure for LLM/Generative AI applications: document chunking, embedding generation pipelines, vector database population, and retrieval layer optimization for RAG architectures.
Requirements
- Degree in computer science or computer engineering