A curated subset of 300 SWE-bench instances designed to be more self-contained and easier to evaluate, while still testing real-world coding ability.
Smaller verified subset is faster to evaluate and iterate on during agent development; still grounded in real repository tasks with executable validation.
Like the full version, scores depend on the agent harness. The curated selection may not represent the full distribution of real-world issues.
Higher is better. Percentage of Lite instances resolved. Top agents score 55-65%.
Max score: 100
Based on ~300 samples, ~5,000 input tokens and ~4,000 output tokens per sample (9,000 total per sample).
Estimates from SWE-bench Lite. Actual costs vary with reasoning effort and multi-turn prompting.
Like the full version, scores depend on the agent harness.
| # | Model | Score | Est. Eval Cost | Score/$ |
|---|---|---|---|---|
| 1 | 80.1 | $21.75 | 3.68 | |
| 2 | 79.5 | $4.50 | 17.7 | |
| 3 | 78.0 | $0.0000 | — | |
| 4 | 77.2 | $4.35 | 17.7 | |
| 5 | 77.2 | $4.50 | 17.1 | |
| 6 | 76.2 | $13.88 | 5.49 | |
| 7 | 75.8 | $2.64 | 28.7 | |
| 8 | 75.4 | $13.88 | 5.44 | |
| 9 | 75.2 | $0.0000 | — | |
| 10 | 75.1 | $13.88 | 5.41 | |
| 11 | 74.4 | $13.88 | 5.36 | |
| 12 | 74.3 | $261.00 | 0.28 | |
| 13 | 74.0 | $21.75 | 3.40 | |
| 14 | 73.3 | $5.22 | 14.0 | |
| 15 | 72.5 | $3.90 | 18.6 | |
| 16 | 71.9 | $0.0000 | — | |
| 17 | 71.7 | $4.35 | 16.5 | |
| 18 | 71.6 | $19.43 | 3.69 | |
| 19 | 71.6 | $17.40 | 4.11 | |
| 20 | 71.3 | $22.50 | 3.17 | |
| 21 | 71.1 | $5.22 | 13.6 | |
| 22 | 70.9 | $0.0000 | — | |
| 23 | 70.1 | $0.0000 | — | |
| 24 | 68.6 | $12.60 | 5.44 | |
| 25 | 68.4 | $13.88 | 4.93 | |
| 26 | 67.7 | $19.43 | 3.48 | |
| 27 | 66.6 | $8.10 | 8.23 | |
| 28 | 66.5 | $10.20 | 6.52 | |
| 29 | 66.3 | $0.0000 | — | |
| 30 | 65.7 | $17.40 | 3.78 | |
| 31 | 65.5 | $13.88 | 4.72 | |
| 32 | 65.3 | $0.0000 | — | |
| 33 | 64.8 | $19.43 | 3.34 | |
| 34 | 64.7 | $4.44 | 14.6 | |
| 35 | 63.9 | $1.88 | 34.1 | |
| 36 | 63.9 | $3.90 | 16.4 | |
| 37 | 63.8 | $0.0000 | — | |
| 38 | 63.7 | $0.0000 | — | |
| 39 | 63.3 | $0.0000 | — | |
| 40 | 63.0 | $17.40 | 3.62 | |
| 41 | 62.8 | $0.15 | 410 | |
| 42 | 62.6 | $0.09 | 674 | |
| 43 | 62.6 | $13.88 | 4.51 | |
| 44 | 62.6 | $0.0000 | — | |
| 45 | 62.6 | $13.88 | 4.51 | |
| 46 | 62.5 | $0.0000 | — | |
| 47 | 62.4 | $0.92 | 67.5 | |
| 48 | 61.7 | $0.51 | 121 | |
| 49 | 61.3 | $21.75 | 2.82 | |
| 50 | 60.6 | $19.43 | 3.12 |