Table of Contents
The most reliable way to compare LLM performance is to test models on the work your application actually performs. Public benchmarks can help you create a shortlist, but they cannot tell you which model will follow your instructions, handle your edge cases, meet your latency target, or deliver the lowest cost per successful task.
A useful evaluation combines application-specific metrics, representative test cases, repeated trials, automated checks, calibrated human review, and production-relevant cost and latency measurements. The goal is not to identify one universally "best" model. It is to find the model and deployment configuration that meet your requirements with acceptable consistency and operating cost.
Choose Evaluation Metrics That Match Your Application
Start by defining what success means for your product. A customer-support assistant, extraction pipeline, coding agent, and summarization tool should not be graded with the same metrics.
Application | Primary Metric | Supporting Metrics | Common Mistake |
|---|---|---|---|
Classification | Accuracy, precision, recall, or F1 | Invalid-label rate and latency | Using accuracy when classes are highly imbalanced |
Structured extraction | Field-level accuracy | Schema validity and missing-field rate | Passing valid JSON that contains incorrect values |
Question answering | Task success or rubric score | Factuality, relevance, and citation support | Treating word overlap as correctness |
Summarization | Coverage and factual consistency | Concision and instruction compliance | Rewarding longer summaries automatically |
RAG | Answer correctness | Groundedness, retrieval recall, and citation accuracy | Grading the answer without checking retrieved evidence |
Coding | Test pass rate | Regression rate, security checks, and maintainability | Accepting code that passes one test but breaks existing behavior |
Agents | End-state success | Tool-call validity, policy compliance, recovery rate, latency, and cost | Grading only the final message |
Creative content | Rubric score or pairwise preference | Brand fit, constraints, and human acceptance | Relying on exact-match metrics |
A strong metric changes a real decision. If improving the metric would not make the application more useful, safer, faster, or less expensive, it probably should not be a release metric.
For deterministic tasks, calculate the accepted-run rate as:
Accepted-Run Rate = Number of Accepted Runs ÷ Total Number of Attempted Runs
Do not exclude malformed responses, timeouts, or tool failures simply because they are inconvenient to grade. If users would experience the event as a failed task, it normally belongs in the denominator.
Open-ended tasks need a scoring rubric. Instead of asking whether an answer is "good," define separate dimensions such as:
Correctness
Completeness
Relevance
Instruction compliance
Groundedness
Tone or style
Safety
Give each dimension clear pass conditions or scoring anchors. "Score factuality from 1 to 5" is too vague. A better rubric explains what evidence separates a 3 from a 4 and identifies any error that causes an automatic failure.
For agent workflows, inspect both the execution trace and final outcome. An agent may claim that it issued a refund, updated a file, or completed a booking even when the underlying state did not change. Outcome checks should therefore inspect the database, file system, application state, or other relevant environment whenever possible.
Repeated trials reveal another important distinction:
pass@k measures whether at least one of kk attempts succeeds.
pass^k measures whether every one of kk attempts succeeds.
Use pass@k when several attempts are acceptable and one successful result is enough. Use pass^k or a per-trial success rate when users expect the workflow to work reliably every time.
Build a Test Set From Real Tasks and Known Failures
An effective LLM evaluation dataset should resemble production traffic without becoming a random sample of everything users have ever submitted.
Start with four sources:
Common production tasks with meaningful usage volume
High-impact workflows tied to revenue, trust, or safety
Edge cases involving unusual formats, missing context, or conflicting instructions
Known failures from support tickets, logs, bug reports, and manual testing
Each test case should contain enough information to reproduce and grade the task:
Not every task needs a single reference answer. Reference answers work well for classification, extraction, and questions with one verifiable result. For open-ended tasks, expected behavior and explicit acceptance criteria are usually more reliable than demanding a specific sentence.
Your dataset should also include paired cases that test both sides of a behavior. If an agent should escalate suspicious activity, include cases where escalation is required and cases where it is unnecessary. Otherwise, the model may appear successful simply by escalating everything.
Keep development and holdout sets separate. Use the development set while changing prompts, tools, or routing logic. Reserve the holdout set for final comparisons so repeated tuning does not turn the evaluation into a memorization exercise.
Version the dataset and record where each case came from. When a production failure appears, add a minimized reproduction to the regression suite. This turns real failures into permanent checks instead of one-time fixes.
Combine Automated Checks, Human Review, and LLM Judges
No single grader works for every output. Use the cheapest reliable method for each criterion, then add more subjective review only where necessary.
Grading Method | Best Uses | Main Limitation |
|---|---|---|
Exact match or regex | Labels, IDs, required fields, and fixed formats | Rejects valid variations |
JSON structure, data types, and required keys | Cannot confirm semantic correctness | |
Unit or integration tests | Code, calculations, tools, and state changes | Requires a testable environment |
Rule-based checks | Length, prohibited content, citations, and tool arguments | Rules may miss nuanced failures |
LLM judge | Relevance, completeness, style, and open-ended quality | Can be biased or inconsistent |
Human review | Ambiguous, high-risk, and expert-dependent tasks | Slow and expensive |
Run deterministic checks first. There is no reason to pay an LLM judge to review an output that is invalid JSON or failed a required unit test.
Use LLM judges for narrowly defined criteria rather than asking for one overall impression. Give the judge the task, relevant context, candidate output, rubric, and permitted labels. Allow an unknown or insufficient evidence result when the judge cannot decide confidently.
Pairwise comparisons can be easier than absolute scoring, but answer order must be controlled. LLM judges may prefer a response because of its placement rather than its quality, a documented form of position bias. Run important pairwise comparisons in both orders and investigate results that reverse when the positions change.
Human reviewers should calibrate the automated system. Have qualified reviewers independently grade a sample, resolve disagreements, and compare their decisions with the LLM judge. Continue sampling after launch, especially for new failure categories and high-impact tasks.
Define task-specific evaluation criteria before running the comparison. Changing the rubric after seeing which model wins introduces bias and makes the reported result difficult to reproduce.
Run a Fair Comparison Across Models
A fair comparison changes the model while keeping the rest of the evaluation controlled.
Keep these elements consistent unless the experiment is specifically testing them:
System and user prompts
Supplied context
Tool definitions
Retrieval results
Maximum output allowance
Timeout policy
Retry policy
Test-case order
Grading logic
Number of trials
Concurrency conditions
Use the exact model identifier for each deployment. Model names, supported parameters, and available versions can differ between an upstream developer, SiliconFlow, another API provider, and a self-hosted deployment. Populate your evaluation configuration with IDs returned by the SiliconFlow model-list endpoint.
Do not force every model to use an unsupported parameter merely to make the request bodies identical. Hold supported settings constant and document any required model-specific difference. The comparison should reflect a valid configuration for every candidate.
Run multiple trials because LLM outputs are nondeterministic. Interleave models instead of running all trials for one model before moving to the next. This reduces the chance that temporary load, network conditions, or another time-dependent factor favors one candidate.
Blind human reviewers and LLM judges to model identity. For pairwise judging, randomize the displayed order and repeat important comparisons with the order reversed.
Retries also need explicit treatment. SDK-level automatic retries can hide multiple physical requests behind one recorded result. Disable them during controlled tests or log every attempt separately. Otherwise, latency, error rate, and cost may be understated.
The following Python example runs repeated trials through SiliconFlow's OpenAI-compatible API. It records output, end-to-end latency, token usage, API failures, and a simple automated acceptance result.
The string checks are intentionally basic. Replace them with graders that match your application.
The documented Chat Completions usage fields provide prompt, completion, and total token counts. Store the raw result records before aggregating them so you can inspect individual failures and recalculate metrics later.
Measure Latency and Cost per Successful Task
A model that produces strong answers but frequently times out, needs several retries, or consumes far more tokens may not be the best production choice.
Measure latency from the user's perspective:
Time to first token: Useful for streaming interfaces
Time to last token: Total generation time
End-to-end latency: Includes retrieval, tools, validation, and other application work
Accepted-task latency: Time required to produce a result that meets the acceptance criteria
Report latency distributions rather than averages alone. Median latency describes a typical request, while p95 or p99 reveals slow-tail behavior. Percentiles require enough trials to be meaningful, so include raw counts when the test set is small.
For agent evaluations, record both total task time and the time spent in each model or tool call. This helps distinguish slow reasoning from slow retrieval, external APIs, browser actions, or repeated recovery attempts.
For models priced per million tokens, calculate request cost as:
Request Cost = (Input Tokens ÷ 1,000,000 × Input Price per Million Tokens) + (Output Tokens ÷ 1,000,000 × Output Price per Million Tokens)
If cached input has a separate rate, add it as another term using the same calculation. Use prices for the exact SiliconFlow deployment being tested rather than rates published by the upstream model developer or another provider.
For repeated trials, use:
Cost per Accepted Task Run = Total Cost Across All Attempted Runs ÷ Number of Accepted Runs
"Total cost" should include failed attempts, retries, invalid outputs, grader calls, retrieval, and paid tool use when those expenses are part of the production workflow. The denominator should contain only runs that meet the predefined acceptance criteria.
If no run is accepted, cost per accepted task is undefined rather than zero. Report the total spend and zero accepted runs instead of producing a misleading cost figure.
This metric often changes the ranking. A cheaper model may cost more per accepted task if it needs frequent retries or fails expensive workflows. A higher-priced model may be more economical when it completes the task reliably on the first attempt.
Set Release Thresholds and Rerun Evaluations After Changes
Evaluation results become operational only when they affect release decisions. Define thresholds before reviewing the final model comparison.
A practical release gate can include:
Release Check | Threshold |
|---|---|
Critical-task failures | Zero allowed |
Overall accepted-run rate | Meets the product target |
Known regression cases | No failures |
p95 end-to-end latency | Within the application limit |
Cost per accepted task | Within the operating budget |
Human and LLM judge agreement | Meets the calibrated target |
Safety or policy violations | Within the approved risk limit |
Avoid hiding a serious regression inside one weighted score. A model should not compensate for unsafe tool behavior by performing slightly better on writing style.
Compare results by task as well as in aggregate. A model may improve the overall average while becoming worse on a small but commercially important category. Report the number of wins, losses, ties, failures, and ungraded cases for every major task group.
For small datasets, do not overinterpret minor percentage differences. Review the individual disagreements and use confidence intervals or paired resampling when the decision is close.
Rerun the evaluation after changes to:
Model or deployment
System prompt
User-prompt template
Sampling or reasoning settings
Retrieval pipeline
Tool definitions
Output schema
Guardrails
SDK or orchestration code
Retry, timeout, or routing logic
Maintain separate capability and regression suites. The capability suite should contain difficult tasks that reveal where the system can improve. The regression suite should protect behavior that already works and normally have a much stricter release threshold.
The final model decision should account for quality, consistency, latency, and cost together. Keep the raw results and evaluation configuration so the comparison can be rerun when models, prices, prompts, or production requirements change.
FAQs
How Large Should an LLM Evaluation Dataset Be?
There is no universal minimum. Start with representative common tasks, high-impact cases, and known failures. Expand the dataset as new production failures appear. A smaller, well-defined set is more useful than a large collection of ambiguous prompts.
Should Public Benchmarks Be Used to Compare LLM Performance?
Yes, but only for initial screening. Public benchmarks measure predefined capabilities under specific conditions. Your final decision should come from an LLM evaluation dataset that reflects your prompts, context, tools, acceptance criteria, latency requirements, and budget.
Can One LLM Judge Grade Every Candidate Model?
It can, but the results should not be accepted without calibration. Compare the judge with qualified human reviewers, test for order sensitivity, permit uncertain results, and use deterministic checks wherever the output can be verified directly.
How Should Infrastructure Failures Be Counted in Agent Evaluations?
Count deployment-specific failures against the tested configuration. Label shared evaluation-infrastructure failures separately when they affect every candidate equally. Preserve both categories so you do not mistake a faulty test environment for weak model performance.
Can API and Self-Hosted Models Be Compared in the Same Evaluation?
Yes, if each candidate receives equivalent tasks and its complete deployment is measured. Record hardware, quantization, serving configuration, retries, concurrency, and provider-specific behavior because the evaluation compares operational systems, not model weights in isolation.
