LLM Evaluation Engineer

Job summary

Build automated evaluation and scoring workflows for LLMs, retrieval systems, and prompt configurations, including synthetic test dataset generation and regression testing. Analyze benchmark datasets, retrieval performance, and production telemetry to assess answer quality, hallucinations, guardrail compliance, and fallback rates.

Job details

Overall – 8+ | AI Exp – 3+ Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure: Context Precision & Context Recall (retrieval quality from vector stores). Faithfulness / Groundedness (checking against hallucination in bounded context). Answer Relevance and Toxicity / Guardrail Compliance. Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends. Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems. Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation. PyTest / Unittest: Writing automated unit/regression tests for prompt templates and model version upgrades. Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL). Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates. Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.

What you’ll do

Build automated evaluation and scoring workflows for LLMs, retrieval systems, and prompt configurations, including synthetic test dataset generation and regression testing. Analyze benchmark datasets, retrieval performance, and production telemetry to assess answer quality, hallucinations, guardrail compliance, and fallback rates.

Requirements

Requires 8+ years of overall experience and 3+ years of AI experience, with hands-on proficiency in LLM evaluation frameworks and APIs, orchestration libraries, and automated testing tools. The role also requires experience with evaluation metrics, complex SQL queries, vector stores, telemetry data, and structured benchmark formats such as JSON, YAML, and CSV.

Listed skills

• SQL · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• LLM Evaluation
• Ragas
• DeepEval
• TruLens
• Promptfoo
• OpenAI Evals
• Retrieval Quality Evaluation
• Faithfulness and Groundedness Evaluation
• Classification Metrics
• Azure OpenAI SDK
• OpenAI API
• LangChain
• LlamaIndex
• PyTest
• SQL
• Vector Stores

Job areas

• Technology
• Software
• Data & Analytics
• Engineering

More jobs from HCLTech