- Eagan, Minnesota, United States
- Full-time · Senior
- 5–7 years
Experienced lawyer focused on improving CoCounsel Legal by writing prompts, experimenting with LLMs, and evaluating AI results to ensure legally sound outputs for legal professionals.
Responsibilities
- Create, test, and refine prompts, skills, and agent configurations for CoCounsel Legal to ensure accurate, legally sound, and contextually appropriate responses
- Experiment with Thomson and other LLMs on legal tasks, comparing results to inform which model best fits each use case
- Build gold-standard datasets with SMEs, coordinate evaluations, and design evaluation schemes, including rubrics, metrics, success criteria, and LLM-as-judge prompts
- Write and run scripts, experiments, and automated tests that ensure quality and consistency over large datasets and time periods
- Monitor production performance, analyze failure modes and identify where in the pipeline they originate (prompting, tooling, retrieval, or content), and provide strategic feedback to software engineers and data scientists
Requirements
- Juris Doctor degree or equivalent law degree
- bar admission in a US or Canadian jurisdiction
- Nice to have: experience with legal products and technology
- Nice to have: online legal research and practice tools
- Nice to have: designing evaluations (rubrics, human grading frameworks, LLM-as-judge prompts)