LLM evaluation metrics: a practitioner's guide for 2026
- 11 minutes ago
- 15 min read
Run a small set of complementary metrics rather than chasing every scorer available: retrieval ranking, semantic similarity, faithfulness, task completion and safety. For retrieval-augmented generation (RAG) systems, split the evaluation in two: score retrieval quality and generation quality separately, because a failure in one masks the true cause of a failure in the other.
This is the practical core of llm evaluation metrics work. Most teams over-engineer this step, wiring up a dozen scorers before they have a single golden dataset to validate any of them against. That is backwards. Start with a small, complementary set aligned to what your users actually need, then expand only where the data tells you to.
Before writing a single evaluator, enable these metric classes:
Retrieval ranking — precision@k, recall@k, or NDCG if you run a RAG pipeline.
Semantic similarity — embedding-based scoring to catch paraphrased but correct answers.
Faithfulness/factuality — claim-level checks against retrieved context to catch hallucination.
Task completion — did the system actually resolve the user’s request, not just produce fluent text.
Safety — toxicity, bias, and policy-violation checks appropriate to your domain.
Treat five as a practical ceiling, not a target. LangChain’s evaluation guidance makes the same point: prioritise a small number of high-signal metrics tied to user goals over a sprawling dashboard nobody reads. Pair that shortlist with a curated golden dataset before you trust any single score.
Key Takeaways
Reliable LLM evaluation depends on splitting RAG retrieval from generation, capping your metric set near five high-signal scores, and validating every automated metric against a human-calibrated golden dataset.
Point | Details |
Split RAG evaluation | Score retrieval (precision@k, recall@k, NDCG) and generation (faithfulness, correctness, coverage) separately to isolate root cause. |
Cap metrics near five | Choose retrieval ranking, semantic similarity, faithfulness, task completion, and safety as a practical starting set. |
Overlap metrics have limits | Use BLEU and ROUGE only for narrow tasks like translation or extractive summarisation, not open-ended generation. |
Calibrate LLM judges | Check judge agreement against human labels using Cohen’s kappa before trusting G-Eval or QAG scores in production. |
Build from real traces | Curate a golden dataset from production queries and edge cases rather than relying on public benchmarks alone. |
Partner for continuous monitoring | Sentient Concepts embeds golden datasets and trajectory checks into managed AI operations so evaluation continues after launch. |
Table of Contents
What llm evaluation metrics measure and how to categorise them
LLM evaluation metrics are the quantitative and qualitative measures used to judge whether a model’s output is accurate, relevant, faithful to its source material, and safe to ship. That sounds simple until you try to apply it, because “accurate” means something different for a translation task than for an open-ended customer support reply. The real work is choosing the right measurement for the right failure mode, not memorising every metric that exists.
Three axes are worth separating clearly because they get muddled constantly in practitioner discussions.
Reference-based vs reference-free. Reference-based metrics compare a model’s output against a known correct answer: BLEU and ROUGE for translation and summarisation, exact-match and F1 for question answering. Reference-free metrics judge output quality without a gold answer to compare against, using either the input context (faithfulness checks) or a separate model acting as judge. Reference-free approaches matter more as LLM outputs get more open-ended, since there often is no single “correct” answer to write down in advance.
Retrieval vs generation. In any RAG system, retrieval and generation are two separate systems with two separate failure modes. Retrieval metrics ask: did we fetch the right documents? Generation metrics ask: given those documents, did the model produce a faithful, correct, and complete answer? Conflating the two into one end-to-end score is the single most common mistake teams make, because it hides which half of the pipeline actually broke.
Statistical vs judged. Perplexity, BLEU, and ROUGE are statistical: they count overlapping tokens or measure how “surprised” a model is by its own output. Judged metrics use a human or another LLM to assign a score based on a rubric. Statistical metrics are cheap and reproducible; judged metrics correlate better with what a human actually cares about but cost more and introduce their own variance.
Here is how those classes map to concrete tools and tasks:
BLEU/ROUGE — n-gram overlap, useful for translation and extractive summarisation, weak for open-ended generation.
Perplexity — measures how well a language model predicts held-out text; useful during pretraining and fine-tuning comparisons, largely useless for judging a single chatbot reply.
BERTScore — embedding-based similarity between reference and candidate, tolerant of paraphrasing.
MRR / NDCG — ranking quality metrics for retrieval, telling you whether the right document showed up near the top of the results.
Accuracy/F1 — for classification-style tasks with a clear correct label, such as intent detection or entity extraction.
G-Eval and other judged metrics — an LLM scores output against a custom rubric, used where no single reference answer exists.
Pro Tip: Map each metric class to a specific failure mode before you write any evaluation code. If you can’t name the failure a metric would catch, it doesn’t belong in your pipeline yet.
Microsoft’s evaluation guidance recommends pairing each metric with a scoring type, whether boolean (pass/fail), categorical (a small set of labels), or continuous (a 0 to 1 score), because the structured feedback types determine how easily you can set thresholds and run regression tests later. A continuous faithfulness score of 0.83 is meaningless without a defined pass threshold; a boolean “hallucinated: yes/no” label is immediately actionable.
How BLEU, ROUGE and perplexity work, and where they break down
BLEU and ROUGE are counting exercises. BLEU checks what proportion of word sequences (n-grams) in a machine-generated translation also appear in a human reference translation. ROUGE flips the direction, checking how much of the reference’s content appears in the generated summary, which is why it became the default for summarisation research. Perplexity measures something different again: how surprised a language model is by the next token in a sequence, with lower perplexity meaning the model assigned higher probability to what actually came next.
These metrics still earn their keep in narrow, well-defined tasks. Machine translation, where there’s a reasonably constrained space of acceptable outputs, is where BLEU was built for and still performs adequately. Extractive or near-extractive summarisation, where the correct output closely mirrors specific phrases from the source, is where ROUGE gives genuine signal, and it remains a standard baseline reported in nearly every summarisation paper published.
The trouble starts the moment output gets more open-ended. A chatbot answer that’s factually correct but phrased completely differently from a reference will score poorly on ROUGE, penalising the model for being fluent rather than repetitive. A summary can score a respectable ROUGE result while completely fabricating a number that never appeared in the source document, because ROUGE only checks word overlap, not truth.
Three failure patterns show up constantly in practice:
A paraphrased but perfectly accurate answer scores low simply because it shares few surface words with the reference.
A fluent, confident-sounding hallucination scores fine on overlap metrics because the fabricated claim happens to share vocabulary with the source text.
Perplexity improves during fine-tuning even as the model’s real-world helpfulness gets worse, because perplexity measures fit to a training distribution, not usefulness to a user.
A widely cited limitation is that ROUGE and similar overlap metrics were designed for summarisation and translation specifically, and their creators never claimed they’d generalise to evaluating open-ended, conversational LLM output. Using them as your primary quality gate for a customer-facing chatbot is applying a ruler built for a different job.
None of this means retire BLEU and ROUGE entirely. Keep them where the task genuinely constrains the output space, and treat any score on an open-ended task as, at best, a weak sanity check rather than a verdict.
Semantic and embedding-based scorers: BERTScore, MoverScore and cosine similarity
Embedding-based scorers solve the paraphrase problem that sinks token-overlap metrics. Instead of counting matching words, they convert both the reference and the candidate text into dense vector embeddings and measure how close those vectors sit in semantic space, usually with cosine similarity. Two sentences that share almost no vocabulary but mean the same thing will still score highly, because the embedding model captures meaning rather than surface form.
BERTScore is the most widely adopted of these. It computes token-level embeddings using a pretrained transformer, matches each token in the candidate to its most similar token in the reference (and vice versa), and reports precision, recall, and F1 from those matches. The precision/recall framing gives you more diagnostic granularity than a single overlap number: low precision suggests the candidate includes content the reference doesn’t support, while low recall suggests the candidate is missing content the reference contains.
MoverStore and its variants take a related but more computationally involved approach, using earth mover’s distance to measure the minimal “cost” of transforming the candidate’s embedding distribution into the reference’s. Cosine similarity on sentence-level embeddings is the lightest-weight option of the three, useful as a fast filter before running anything more expensive.
Choose embedding scorers over overlap metrics whenever your outputs have legitimate variation in phrasing: chatbot responses, open-ended question answering, and abstractive summarisation all fall into this category. Combine both types when you need a fast, cheap first pass (overlap metrics) plus a more nuanced second check on the outputs that pass the first filter.
The trade-offs are real and worth taking seriously before you lean on embedding scores as a primary quality gate. Research on embedding-based metrics for natural language generation found they correlate poorly with human judgement on tasks that require subtle reasoning, logical consistency, or factual precision, since semantic closeness in embedding space doesn’t guarantee the candidate is actually correct, only that it’s topically similar. Two answers can sit close together in embedding space while one contains a critical factual error the other doesn’t.
Three further caveats deserve attention:
Calibration — a BERTScore of 0.87 means nothing in isolation; you need a baseline distribution from known-good and known-bad outputs before you can set a meaningful threshold.
Dataset bias — the underlying embedding model carries whatever biases it learned during pretraining, which can systematically favour or penalise certain phrasings, dialects, or writing styles.
Interpretability — a low embedding score tells you something is off, but rarely tells you what, which slows down debugging compared with a claim-level faithfulness check.
Splitting RAG evaluation into retrieval and generation
Retrieval-augmented generation systems fail in one of two distinct places, and lumping them into a single end-to-end score hides which one broke. Evaluate retrieval and generation as separate systems with separate metrics, then correlate the results afterwards to diagnose root cause.
Retrieval evaluation asks a narrow question: given a query, did the system fetch the right supporting documents? You need a golden dataset of queries paired with the document chunks that should be retrieved for each one, then compute:
Precision@k — of the top k chunks retrieved, what proportion are actually relevant.
Recall@k — of all relevant chunks that exist, what proportion made it into the top k.
Mean Reciprocal Rank (MRR) — how high up the ranking the first relevant chunk appears, penalising systems that bury the right answer on page two.
NDCG (Normalised Discounted Cumulative Gain) — rewards systems that rank the most relevant chunks highest, not just present somewhere in the list.
Generation evaluation asks a different question: given the retrieved context, did the model produce a response that’s faithful, correct, and complete? Three metrics matter most here. Faithfulness checks whether every claim in the answer is actually supported by the retrieved context, catching hallucination directly. Correctness checks the answer against ground truth where one exists. Coverage checks whether the answer captures everything the retrieved context says that’s relevant to the question, catching incomplete answers that technically don’t contain false statements but still fail the user.
The strongest practical technique for faithfulness is claim decomposition: break the generated answer into individual factual assertions, then check each one independently against the source context rather than judging the whole answer holistically. Cohere’s deep dive into RAG evaluation describes this claim-by-claim approach as far more precise at pinpointing exactly which sentence in a response is unsupported, rather than issuing a single vague “this seems unfaithful” verdict on the entire output.
Open RAG Eval’s metrics documentation formalises this split explicitly, treating retrieval and generation as separate scoring surfaces and recommending faithfulness checks specifically to catch hallucination that overlap or embedding metrics would miss entirely. Aggregate scores across your full evaluation set, but always keep per-query results available. A system with a strong average faithfulness score can still be failing badly on a specific query category, such as multi-hop questions that require synthesising information across several retrieved chunks, and that pattern only shows up when you inspect individual queries rather than trusting the headline number.
Pro Tip: When faithfulness scores drop, check retrieval precision first. A generation model faithfully repeating irrelevant retrieved content will still fail a faithfulness check, and the fix is in your retriever, not your prompt.
LLM-as-a-judge: G-Eval, DAG and QAG in practice
LLM-as-a-judge approaches use a separate model, usually a stronger or differently-tuned one, to score outputs against a written rubric rather than comparing against a fixed reference answer. This matters wherever a single “correct” answer doesn’t exist, which describes most real customer-facing generation tasks.
G-Eval prompts an LLM with a detailed scoring rubric and a chain-of-thought reasoning step, asking it to explain its reasoning before assigning a numeric score. This tends to produce more consistent scores than asking for a bare number, because the reasoning step forces the judge model to engage with specifics rather than pattern-matching to a vague overall impression.
DAG (Decision-tree Aggregated Grading) breaks scoring into a tree of yes/no or categorical sub-decisions rather than one holistic judgement. Instead of asking “is this answer good?”, DAG asks a sequence of narrower questions such as “does the answer address the question asked?” then “is every claim supported by the context?” then “is the tone appropriate?”, combining the sub-scores at the end. This decomposition makes failures far easier to trace than a single opaque score.
QAG (Question-Answer Generation) works differently again: it generates a set of factual questions from the reference or source material, has the model-under-test’s output answer those same questions, then checks agreement between the two answer sets. It’s particularly effective for factuality checks because it turns a fuzzy “is this faithful?” judgement into a series of concrete, checkable question-answer pairs.
Building a reliable judge takes real calibration work, not just a well-written prompt. Confident AI’s evaluation guide recommends running the judge against a human-labelled calibration set before trusting it in production, then measuring agreement using Cohen’s kappa or Krippendorff’s alpha between the judge and your human annotators. Low agreement means your rubric needs rewriting, not that you should simply trust the judge anyway.
Practical steps that reduce judge instability:
Write the rubric with concrete examples of a 1, 3, and 5 score, not just abstract descriptions.
Run the same evaluation with multiple random seeds or at a slightly higher temperature and check for consensus rather than trusting a single pass.
Recalibrate periodically against fresh human labels, since concept drift in your production traffic can silently shift what “good” looks like.
Use a stronger, more capable model as judge than the model being evaluated wherever budget allows.
Pro Tip: Never let the same model judge its own outputs without a calibration check first. Self-preference bias is well documented, and a model grading its own family’s responses tends to score them more generously than an independent judge would.
Judge-based scores earn their place when you’re evaluating open-ended quality dimensions such as helpfulness, tone, or reasoning coherence, where no fixed reference could capture every acceptable answer. Combine them with reference-based or retrieval metrics wherever a ground truth genuinely exists; there’s no reason to pay for an expensive judge call to check something a simple exact-match comparison would catch just as reliably.
Building the evaluation pipeline: from golden dataset to production monitoring
Treat evaluation as a pipeline with three connected stages, not a one-off test you run before shipping. Each stage catches problems the others miss.
Build a golden dataset from production traces. Sample real queries your system has actually received, not synthetic examples you imagined in advance. Curate deliberately for edge cases: ambiguous questions, multi-hop queries, adversarial phrasing, and known failure categories from support tickets. A golden dataset of 150 to 300 carefully chosen examples, annotated by someone who understands the domain, outperforms a generic 10,000-example public benchmark for catching your system’s actual failure modes.
Run offline experiments before every release. Score every candidate model or prompt change against the full golden dataset, including deliberate stress tests, adversarial prompts designed to induce hallucination, and edge cases pulled from past incidents. Treat any metric regression against your last known-good baseline as a release blocker, not a footnote.
Add trajectory checks for anything agentic. If your system chains tool calls or takes multiple reasoning steps, score the full path, not just the final answer. A correct final answer reached via a broken or inefficient reasoning path is a fragile success that will fail differently next time.
Instrument continuous production monitoring. Track metric drift over time, not just point-in-time scores, since a faithfulness score quietly declining over three weeks is a different problem than a sudden cliff after a model provider update.
Public benchmarks have a real but limited role here. They’re genuinely useful for shortlisting candidate models before you commit engineering time to integration, but benchmark performance rarely reflects production failure modes specific to your domain and data. Use benchmarks to narrow your options, then validate the finalists against your own golden dataset before making a final call.
Production monitoring needs its own signals distinct from your offline test suite:
Sample a percentage of live traffic for automated scoring plus periodic human spot-checks, since combining metric-level drift alerts with sampled human review catches subtle quality degradation that automated metrics alone tend to miss.
Set alert thresholds based on your calibration data, not arbitrary round numbers.
Track cost and latency alongside quality metrics, since a faithfulness improvement that triples response time is rarely a net win in production.
Log full input, output, and retrieved context for every flagged interaction, so a human reviewer can actually diagnose what went wrong.
The 5-metric rule and how Sentient Concepts applies it
The strongest heuristic in this entire field is deceptively simple: pick five metrics, or fewer, and make every one of them earn its place. Teams that instrument twenty scorers rarely act on more than two or three of them consistently, and the unused seventeen just add noise, cost, and false confidence to a dashboard nobody actually reads before shipping.
Concrete mixes that hold up across common use cases:
RAG question answering — retrieval precision@k, faithfulness (claim-level), answer correctness, and a safety/toxicity check.
Summarisation — ROUGE as a cheap first-pass filter, BERTScore for semantic coverage, and a judge-based coherence score for the outputs that pass both.
Agentic workflows — task completion rate, trajectory correctness (did the tool-call sequence make sense), and a safety check on any action with real-world side effects.
Sentient Concepts builds this discipline directly into engagements, because a metric that never gets reviewed is worse than no metric at all. That starts with a golden dataset curated from a client’s actual production traces rather than a generic public benchmark, moves through trajectory checks for any agentic component, and continues into managed AI operations once a system goes live, so evaluation doesn’t stop at launch day.
The decision flow that actually works in practice is: detect the failure mode you’re worried about first, then pick the smallest set of metrics that would catch it, then set a threshold calibrated against real examples, and only then automate monitoring. Teams that reverse this order, instrumenting metrics before naming a specific failure they’re guarding against, end up with dashboards full of numbers nobody can act on.
That sequencing, failure mode before metric, metric before threshold, threshold before automation, is the difference between an evaluation pipeline a team actually trusts and one that just looks thorough in a slide deck.
What the research actually supports
Most advice on this topic reads like a metrics catalogue: here are forty scorers, pick whichever sound relevant. That framing is backwards, and it produces evaluation suites nobody trusts enough to act on. The research consistently points the other way: fewer metrics, chosen against named failure modes, checked regularly against human judgement, beats comprehensive coverage every time.
The most underrated point in all of this is calibration. Teams will happily build a BERTScore or G-Eval pipeline and never once check it against a human-labelled set to see whether it actually agrees with what a person would say. An uncalibrated metric isn’t just useless, it’s actively dangerous, because it creates false confidence that masks real quality problems until a customer notices first.
If you take one thing from this, take the RAG split. Conflating retrieval and generation into one score is the single most common mistake, and it’s also the easiest one to fix. Separate them, calibrate your judges, and keep the metric count small enough that someone actually looks at it every week.
Turning evaluation strategy into a running pipeline
Knowing which metrics to run is one problem. Keeping them running reliably against live traffic, recalibrated as your data drifts, alerted correctly when something breaks, is a different and ongoing one. Sentient Concepts exists for teams that would rather hand off that operational weight than build and staff it internally, and does so without the handoffs that typically stall evaluation work between a strategy phase and an engineering team.
If your team has working models but no golden dataset, no trajectory checks on agentic components, and no alerting when faithfulness drifts, that gap is exactly where evaluation pipelines quietly fail in production. Smaller teams often keep this in-house successfully with a disciplined five-metric approach; teams running multiple production LLM systems across departments tend to benefit from a partner who has already built the calibration and monitoring infrastructure. Sentient Concepts’ managed AI operations service builds the golden dataset, sets up trajectory checks, and runs continuous monitoring with defined alert thresholds, carrying the same accountability from the initial strategy work through to day-to-day operation. Get in touch to discuss what a working evaluation pipeline would look like for your specific systems.
Sources
Building your own evaluators benefits enormously from reading the primary documentation behind each metric class rather than relying on secondhand summaries. These are worth bookmarking:
FAQ
How can I evaluate the performance of an LLM?
Combine reference-based metrics where a correct answer exists, embedding-based or LLM-as-a-judge scoring for open-ended tasks, and always validate results against a small, human-calibrated golden dataset before trusting the numbers in production.
What are AI evaluation metrics?
AI evaluation metrics are quantitative and qualitative measures used to judge model output quality, spanning token-overlap scores like BLEU and ROUGE, embedding-based similarity like BERTScore, and judge-based rubric scoring like G-Eval.
What are LLM evaluation frameworks?
LLM evaluation frameworks are structured pipelines that combine golden datasets, offline testing, trajectory checks for multi-step agents, and online production monitoring, rather than a single metric run once before release. Sentient Concepts builds this structure directly into its managed AI operations engagements.
What are the metrics used in LLM-as-a-Judge evaluation?
The three main approaches are G-Eval, which uses chain-of-thought reasoning before scoring against a rubric; DAG, which breaks judgement into a decision tree of narrower sub-questions; and QAG, which generates factual questions and checks answer agreement to test faithfulness.
How many metrics should I actually track?
A practical ceiling is around five complementary metrics covering retrieval, semantic accuracy, faithfulness, task completion, and safety, since teams that instrument far more than this rarely act on the extra scores consistently.
Recommended