LLM Evaluation Metrics: What to Measure Beyond Accuracy
LLM evaluation needs metrics that reflect the use case. This guide covers correctness, completeness, groundedness, harmfulness, human review and LLM-as-a-judge.
By Maya Chen, Women in AI Editorial Fellow ยท 30 August 2026
There is no single LLM evaluation metric that tells you whether a generative AI application is good.
The useful metric depends on the task. A factual research assistant, creative-writing tool and customer-service agent should not be judged in the same way.
The first engineering job is therefore to define what quality means for the product.
Correctness
Correctness asks whether the output is factually or logically right.
This is essential for systems that answer questions, perform calculations or extract information. It can be measured against reference answers, structured rules or expert review.
The limitation is that many valid generative responses do not match one exact reference.
Completeness
An answer can be correct but incomplete.
If a user requests five requirements and the model returns three accurately, a pure factuality score may look good while the task still fails.
Completeness is especially important in summarisation, compliance and multi-part instructions.
Groundedness
Groundedness measures whether claims are supported by the source material available to the model.
This is central to RAG systems. An answer can be plausible and even true in the real world but still be ungrounded if it invents information not contained in the provided evidence.
Harmfulness and safety
Safety evaluation looks for outputs that violate policy or create unacceptable risk.
The relevant dimensions vary by use case and may include harmful instructions, discriminatory content, privacy leakage or inappropriate professional advice.
Task success
For applications and agents, task success can be more meaningful than linguistic quality.
Did the system complete the requested workflow? Did it use the right data? Did it leave the system in the correct state?
This moves evaluation closer to the user's actual outcome.
LLM-as-a-judge
A model can be used to score another model's outputs against defined criteria.
AWS and other platforms now expose LLM-as-a-judge evaluation approaches because they scale better than manual review for subjective dimensions such as completeness or helpfulness.
The technique is useful but should itself be validated. Judge models can have bias and inconsistency, so teams should compare automated scores against trusted human assessment.
Human review still matters
For high-impact systems, human evaluation remains valuable because people can notice failure modes the metric design did not anticipate.
A strong evaluation programme usually combines programmatic metrics, model-based evaluators and targeted expert review.
If you are building production AI systems, the Women in AI Global Summit in London will include practitioners working across AI engineering, enterprise deployment and evaluation. The useful conversations are often about the less glamorous part of AI: how teams know a system is actually good enough to trust.
Build the metric around the decision
Every evaluation metric should support a decision.
Does this model outperform the current one? Is the RAG system ready to ship? Did a prompt change reduce hallucinations? Should an agent receive broader permissions?
If a metric cannot help answer a product or engineering decision, it may be measurement without purpose.