DualEval: Joint Model–Item Calibration for Unified LLM Evaluation

A latent calibration framework that fits static benchmark correctness and arena reward scores together — producing one consistent model ranking alongside per-question difficulty and discrimination.

Aaron J. Li¹ · Hao Huang¹ · Youngmin Park¹ · Yitong Ma¹ · Wei-Lin Chiang² · Li Chen² · Cho-Jui Hsieh^2,3 · Bin Yu¹ · Ion Stoica^1,2

¹ University of California, Berkeley · ² Arena · ³ University of California, Los Angeles

arXiv Code Get started

Joint IRT ranking for the Coding domain: model ability estimates plotted against item difficulty distribution — **Joint model–item ranking (Coding domain).** Models are positioned on the latent ability scale (θ, dashed lines); the histogram shows item difficulty (b) from five question sources. Items to the left of a model’s line are those it is expected to answer correctly.

Motivation

Two signals, no bridge. Static benchmarks give objective correctness labels; arenas give reward-model scores over open-ended responses. Both are informative, but existing methods use them in isolation — producing separate leaderboards that often disagree.
Not all questions are equal. A question every model answers correctly carries no discriminative signal. A question that reliably separates strong from weak models is worth far more. Simple accuracy averaging ignores this.
Questions have quality too. Current evaluation ranks models but never asks which questions were informative, which were redundant, or which responses were anomalous.
DualEval unifies both signals. By fitting static and arena data through a shared calibration model, it jointly estimates model ability and per-item difficulty and discrimination — one consistent ranking, with interpretable item-level diagnostics.

Leaderboard

Model rankings by domain

Estimated latent ability θ from DualEval (joint static + arena mode) across 18 frontier models. Coding and Misc each use 300 static questions and 300 arena prompts; Math uses 200 static questions and 200 arena prompts. Higher θ indicates greater estimated ability on that domain’s item pool.

#	Model	Score
1	Claude Opus 4.7	+0.616
2	GPT-5.5	+0.548
3	Claude Opus 4.6	+0.525
4	GPT-5.4	+0.497
5	Gemini 3.1 Pro	+0.429
6	GPT-5.4 Mini	+0.377
7	Claude Sonnet 4.6	+0.238
8	Gemini 2.5 Pro	+0.037
9	GPT-5 Mini	-0.086
10	DeepSeek V3.2	-0.092
11	Claude Haiku 4.5	-0.111
12	Qwen3 Max (Thinking)	-0.135
13	Gemini 2.5 Flash	-0.181
14	GPT-4.1	-0.347
15	GPT-4.1 Mini	-0.372
16	Grok 4	-0.533
17	Mistral Large 3	-0.619
18	Llama 4 Maverick	-0.790

#	Model	Score
1	GPT-5.5	+0.623
2	Claude Opus 4.7	+0.615
3	Gemini 3.1 Pro	+0.530
4	Grok 4	+0.355
5	Qwen3 Max (Thinking)	+0.217
6	GPT-5 Mini	+0.170
7	Gemini 2.5 Pro	+0.132
8	GPT-5.4	+0.075
9	Claude Opus 4.6	+0.037
10	Gemini 2.5 Flash	+0.028
11	Claude Sonnet 4.6	-0.028
12	DeepSeek V3.2	-0.087
13	GPT-4.1 Mini	-0.103
14	Mistral Large 3	-0.360
15	Claude Haiku 4.5	-0.403
16	GPT-4.1	-0.419
17	GPT-5.4 Mini	-0.498
18	Llama 4 Maverick	-0.885

#	Model	Score
1	GPT-5.5	+0.506
2	Gemini 3.1 Pro	+0.441
3	Claude Opus 4.6	+0.364
4	Claude Opus 4.7	+0.327
5	Claude Sonnet 4.6	+0.250
6	Gemini 2.5 Pro	+0.238
7	GPT-5.4	+0.115
8	Grok 4	+0.062
9	Qwen3 Max (Thinking)	+0.018
10	Mistral Large 3	-0.041
11	GPT-5 Mini	-0.064
12	Gemini 2.5 Flash	-0.071
13	Claude Haiku 4.5	-0.183
14	GPT-5.4 Mini	-0.190
15	DeepSeek V3.2	-0.214
16	GPT-4.1	-0.262
17	GPT-4.1 Mini	-0.523
18	Llama 4 Maverick	-0.775

Applications

Model rankings, item profiles, and residual diagnostics

Joint rankings across data sources

Static correctness and arena reward scores update the same latent ability scale. One consistent model ranking instead of two separate leaderboards that you have to reconcile by hand.

Benchmark compression

Discrimination scores identify which questions provide the most separation among your current model pool. Evaluate on the top-k most informative items without meaningfully changing your model order.

Anomaly detection

Residuals from the fitted model flag model-item cells that deviate from predicted behaviour — a principled screen for benchmark contamination, capability outliers, and reward model inconsistencies.

Method

Item Response Theory, extended for LLM evaluation

01 · Static

Two-parameter IRT

Each model i has a latent ability θ_i; each item q has a difficulty b_q and a discrimination a_q. Binary correctness labels are fit through the two-parameter logistic success probability.

p_i,q=σ(a_q(θ_i−b_q))

02 · Arena

Reward-distilled pairwise IRT

Reward scores are normalized and distilled into soft pairwise targets, fit through the same latent parameters by a learned temperature γ. Ties and “both-bad” pairs are handled separately, so only decisive comparisons drive the ranking.

P̂(i≻j|q)=σ(γ(p_i,q−p_j,q))

03 · Joint

Shared-parameter optimization

The static and arena losses are summed over one shared set of parameters, so both signals reinforce the same latent scale, with a regularizer ℛ for stability. Abilities are zero-centered each step for identifiability, and the final ranking simply sorts models by θ.

ℒ=ℒ_static+ℒ_arena+λ_reg ℛ

04 · Outputs

What you get

A model ranking by ability θ, per-item difficulty b and sharpness a profiles, and per‑cell residuals for anomaly screening — all from a single gradient-based fit over shared parameters.

Usage

DualEval reads two JSONL files — supply either or both. Each line is one (model, item) record; the loaders concatenate multiple files and keep item IDs unique per source.

Static labels

One row per (model, question) with a binary correctness label from a benchmark.

{"model_label": "gpt-4o",
 "dataset": "gsm8k",
 "sample_index": 0,
 "correct": 1}

model_label — unique model identifier
dataset + sample_index — together identify the question
correct — 1 if correct, 0 otherwise

Arena rewards

One row per (model, prompt) with a continuous reward-model score.

{"model_label": "gpt-4o",
 "item_id": "arena_math_0042",
 "reward": 3.71}

model_label — unique model identifier
item_id — unique prompt identifier
reward — reward score (any real number)

Rewards are z-normalized automatically and rows missing a required field are skipped, so partial coverage is fine. Point the loaders at your files and fit:

# install
pip install -r requirements.txt

# programmatic API
from dualeval import fit_irt, load_static_jsonl, load_arena_reward_jsonl

static_df = load_static_jsonl(["results/static/math.jsonl"])
arena_df  = load_arena_reward_jsonl(["results/arena/math.jsonl"])

results = fit_irt(static_df=static_df, arena_df=arena_df, mode="both")
print(results["model_ranking"])    # sorted by estimated ability θ
print(results["question_ranking"])  # per-item difficulty and discrimination

# or config-driven CLI
python dualeval/dualeval.py --config my_config.yaml

See the GitHub repository for the full config reference, fitting modes, and the end-to-end reproduction pipeline.

Citation

@article{dualeval2026, title = {DualEval: Joint Model-Item Calibration for Unified LLM Evaluation}, journal = {arXiv preprint arXiv:2606.26429}, year = {2026}, }