Long Context Reasoning — evaluates a model's ability to reason over very long documents, synthesizing information spread across large contexts.
Distinguishes models that can synthesize evidence across distant passages from those that only retrieve local snippets; useful for comparing context utilization as window sizes scale.
Scores can vary significantly depending on where key information is placed in the context window (beginning vs. middle vs. end).
Higher is better. Tests reasoning accuracy at various context lengths.
Max score: 100
Based on ~200 samples, ~800 input tokens and ~1,000 output tokens per sample (1,800 total per sample).
Estimates from LCR benchmark. Actual costs vary with reasoning effort and multi-turn prompting.
Scores can vary significantly depending on where key information is placed in the context window (beginning vs.
| # | Model | Score | Est. Eval Cost | Score/$ |
|---|---|---|---|---|
| 1 | 88.7 | $3.48 | 25.5 | |
| 2 | 88.3 | $0.70 | 126 | |
| 3 | 86.3 | $0.24 | 354 | |
| 4 | 86.3 | $2.32 | 37.2 | |
| 5 | 85.3 | $11.60 | 7.36 | |
| 6 | 84.7 | $11.60 | 7.30 | |
| 7 | 84.7 | $4.64 | 18.2 | |
| 8 | 84.7 | $4.64 | 18.2 | |
| 9 | 84.3 | $1.05 | 80.3 | |
| 10 | 84.3 | $4.64 | 18.2 | |
| 11 | 84.3 | $6.80 | 12.4 | |
| 12 | 84.3 | $6.80 | 12.4 | |
| 13 | 84.0 | $0.87 | 96.6 | |
| 14 | 84.0 | $0.29 | 293 | |
| 15 | 84.0 | $2.32 | 36.2 | |
| 16 | 84.0 | $4.64 | 18.1 | |
| 17 | 83.7 | $11.60 | 7.21 | |
| 18 | 83.7 | $2.32 | 36.1 | |
| 19 | 83.7 | $2.32 | 36.1 | |
| 20 | 83.7 | $0.0000 | — | |
| 21 | 83.7 | $0.27 | 308 | |
| 22 | 83.3 | $0.29 | 289 | |
| 23 | 83.3 | $3.08 | 27.1 | |
| 24 | 83.3 | $0.12 | 718 | |
| 25 | 83.3 | $0.10 | 868 | |
| 26 | 83.3 | $2.32 | 35.9 | |
| 27 | 83.0 | $11.60 | 7.16 | |
| 28 | 83.0 | $1.05 | 79.0 | |
| 29 | 83.0 | $0.0000 | — | |
| 30 | 83.0 | $1.05 | 79.0 | |
| 31 | 83.0 | $0.0000 | — | |
| 32 | 83.0 | $2.32 | 35.8 | |
| 33 | 83.0 | $0.23 | 364 | |
| 34 | 83.0 | $6.80 | 12.2 | |
| 35 | 83.0 | $0.29 | 288 | |
| 36 | 83.0 | $2.72 | 30.5 | |
| 37 | 82.7 | $4.64 | 17.8 | |
| 38 | 82.7 | $2.32 | 35.6 | |
| 39 | 82.7 | $0.12 | 713 | |
| 40 | 82.7 | $1.16 | 71.3 | |
| 41 | 82.7 | $0.29 | 287 | |
| 42 | 82.7 | $3.08 | 26.8 | |
| 43 | 82.3 | $11.60 | 7.10 | |
| 44 | 82.3 | $2.32 | 35.5 | |
| 45 | 82.3 | $2.32 | 35.5 | |
| 46 | 82.3 | $0.23 | 361 | |
| 47 | 82.3 | $1.16 | 71.0 | |
| 48 | 82.3 | $3.08 | 26.7 | |
| 49 | 82.3 | $11.60 | 7.10 | |
| 50 | 82.3 | $4.64 | 17.7 |