Data Scientist III
A hands-on role joining the AMI team combining Machine Learning, statistical modeling, Generative AI, Retrieval-Augmented Generation (RAG), model evaluation, and applied AI experimentation for enterprise AI applications.
- Remote, United States
- Full-time
- Remote
- $132K–$220K per year
- Posted Nov 5, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine Learning
Responsibilities
- Design, develop, evaluate, and improve AI/ML solutions that address enterprise business and technical challenges.
- Apply statistical rigor, reproducible experimentation, and MLOps/LLMOps practices to improve model quality, RAG effectiveness, and AI application performance.
- Take ownership of defined Data Science initiatives, lead moderately complex or ambiguous modeling and evaluation tasks, collaborate with engineering teams on production readiness, and mentor less experienced Data Scientists.
- Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
- Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques using traditional ML, embeddings, and LLM-based approaches.
- Apply statistical modeling to analyze execution latency, token usage, inference costs, workload trends, and resource requirements.
- Use clustering and anomaly detection to identify recurring intents, unusual agent behavior, capability gaps, and execution regressions.
- Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments.
- Evaluate commercial and self-hosted open-weight models for accuracy, instruction adherence, structured outputs, latency, and cost.
- Develop and test prompt templates, few-shot examples, tool-calling prompts, and context assembly approaches.
- Compare model alternatives for defined enterprise use cases and document quality, performance, and cost tradeoffs.
- Collaborate with Senior Data Scientists and AI Engineering on model performance analysis and inference optimization.
- Develop and evaluate document chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
- Compare vector, keyword/BM25, hybrid, and semantic retrieval approaches.
- Measure retrieval effectiveness using Recall@K, Precision@K, nDCG, groundedness, faithfulness, and answer quality.
- Investigate retrieval and generation failures and recommend improvements based on evaluation results.
- Benchmark embedding and reranking models using representative enterprise datasets.
- Support evaluation of prompting, RAG, supervised instruction fine-tuning, and parameter-efficient model adaptation approaches.
- Prepare and validate instruction datasets, training examples, and holdout evaluation datasets.
- Participate in SFT, LoRA, QLoRA, or comparable fine-tuning experiments under appropriate technical guidance.
- Compare adapted models against baseline models for accuracy, generalization, regressions, and operational cost.
- Collaborate on validation and production handoff of fine-tuned model artifacts.
- Apply modeling and evaluation techniques to intent classification, agent/skill discovery, tool selection, and routing effectiveness.
- Evaluate task completion, tool-call correctness, parameter extraction, delegation quality, and recovery behavior.
- Understand agentic AI concepts, including planning, tool calling, MCP tools, A2A Agent Cards, skills, and task delegation.
- Collaborate with AI Engineering to investigate model-related issues affecting agentic workflows.
- Contribute model-based decision signals and evaluation approaches without owning agent runtime, MCP server, or A2A protocol implementation.
- Develop evaluation datasets, golden test cases, holdout sets, and automated evaluation workflows.
- Evaluate ML models, foundation models, prompts, RAG pipelines, fine-tuned models, and agentic AI behaviors.
- Apply statistical analysis, human evaluation, and LLM-as-judge approaches with appropriate calibration.
- Track accuracy, relevance, groundedness, instruction following, task completion, latency, and cost.
- Investigate evaluation failures and recommend improvements based on experimental evidence.
- Support automated regression testing and production readiness validation.
- Apply experiment tracking and versioning across datasets, code, model configurations, prompts, retrieval settings, metrics, and artifacts.
- Develop and maintain reproducible model training, benchmarking, and evaluation workflows.
- Use model registries, experiment tracking tools, and automated evaluation pipelines.
- Contribute to model quality gates, regression testing, and release-readiness checks.
- Analyze model performance, quality drift, routing accuracy, latency, and inference cost signals.
- Collaborate with engineering teams on model packaging, deployment handoffs, monitoring, and lifecycle management.
Requirements
- Bachelor's degree in Data Science, Machine Learning, or Applied AI, Physics, Computer Science, or equivalent
- Master's degree in a related field
- Experience in a Enterprise AI
- Experience in a Commercial AI
Nice to have: Agentic AI, MCP tools, A2A interoperability, LLM-as-judge, Experiment tracking, Quantization, vLLM, Triton, FastAPI, Flask, OpenTelemetry, Docker, OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, LangGraph, OpenAI, CrewAI, Ragas, DeepEval, OpenJudge, LangSmith, MLflow, Kubeflow
Skills
Machine LearningApplied AIStatistical ScienceStatistical modelingGenerative AIRetrieval-Augmented Generation (RAG)MLOpsLLMOpsClassificationRegressionRankingForecastingClusteringAnomaly detectionEmbeddingsLLM-based approachesPrompt EngineeringVector searchSFTLoRAQLoRAPythonPandasNumPyscikit-learnPyTorchTensorFlowSQLPythonPandasNumPyscikit-learnPyTorchTensorFlowSQL
Benefits
- variable plan
- country specific benefits
Full description
Company details are for members
Sign up for a free job alert to see BMC Software's profile, website and LinkedIn, and the same for every employer on Referso Jobs.