SWE-bench Verified is a static, human-filtered subset of real GitHub issue-resolution tasks. AgMoDB preserves its historical results for inspection but no longer uses them in the default AgMoBench composite.
Grounded in real repositories with executable tests; the verified subset removed many ambiguous or broken tasks from the original SWE-bench release.
OpenAI found material test or problem-statement issues in at least 59.4% of a 138-task often-failed audit subset and found contamination evidence across every tested frontier model. Scores are also scaffold-sensitive, so treat them as historical agent-system measurements rather than a clean current model ranking.
Higher is better. Percentage of issues resolved by the full agent system; AgMoDB excludes this benchmark from the default AgMoBench composite.
Max score: 100
Based on ~500 samples, ~5,000 input tokens and ~4,000 output tokens per sample (9,000 total per sample).
Estimates from SWE-bench Verified. Actual costs vary with reasoning effort and multi-turn prompting.
OpenAI found material test or problem-statement issues in at least 59.4% of a 138-task often-failed audit subset and found contamination evidence across every tested frontier model.
| # | Model | Score | Est. Eval Cost | Score/$ |
|---|---|---|---|---|
| 1 | 93.9 | — | — | |
| 2 | 87.6 | $62.50 | 1.40 | |
| 3 | 86.4 | $36.25 | 2.38 | |
| 4 | 85.9 | $0.0000 | — | |
| 5 | 85.8 | $0.0000 | — | |
| 6 | 85.3 | $0.0000 | — | |
| 7 | 85.0 | $0.45 | 189 | |
| 8 | 84.0 | $0.74 | 114 | |
| 9 | 83.0 | $4.40 | 18.9 | |
| 10 | 82.9 | $0.0000 | — | |
| 11 | 82.6 | $0.68 | 122 | |
| 12 | 82.6 | $7.40 | 11.2 | |
| 13 | 81.8 | $210.00 | 0.39 | |
| 14 | 80.9 | $0.0000 | — | |
| 15 | 80.9 | $62.50 | 1.29 | |
| 16 | 80.9 | $62.50 | 1.29 | |
| 17 | 80.9 | $0.85 | 95.2 | |
| 18 | 80.8 | $62.50 | 1.29 | |
| 19 | 80.8 | $62.50 | 1.29 | |
| 20 | 80.6 | $29.00 | 2.78 | |
| 21 | 80.6 | $29.00 | 2.78 | |
| 22 | 80.6 | $29.00 | 2.78 | |
| 23 | 80.3 | $36.25 | 2.21 | |
| 24 | 80.0 | $32.38 | 2.47 | |
| 25 | 80.0 | $32.38 | 2.47 | |
| 26 | 80.0 | $32.38 | 2.47 | |
| 27 | 79.9 | $0.90 | 88.8 | |
| 28 | 79.7 | $5.55 | 14.4 | |
| 29 | 79.6 | $4.63 | 17.2 | |
| 30 | 79.6 | $37.50 | 2.12 | |
| 31 | 79.6 | $37.50 | 2.12 | |
| 32 | 79.6 | $37.50 | 2.12 | |
| 33 | 79.5 | $0.0000 | — | |
| 34 | 79.4 | $187.50 | 0.42 | |
| 35 | 79.4 | $62.50 | 1.27 | |
| 36 | 79.4 | $187.50 | 0.42 | |
| 37 | 79.0 | $37.50 | 2.11 | |
| 38 | 79.0 | $0.0000 | — | |
| 39 | 79.0 | $0.0000 | — | |
| 40 | 78.8 | $0.0000 | — | |
| 41 | 78.7 | $0.93 | 84.6 | |
| 42 | 78.5 | $0.0000 | — | |
| 43 | 78.1 | $29.00 | 2.69 | |
| 44 | 78.1 | $0.0000 | — | |
| 45 | 78.0 | $7.25 | 10.8 | |
| 46 | 78.0 | $7.25 | 10.8 | |
| 47 | 77.2 | $37.50 | 2.06 | |
| 48 | 77.2 | $37.50 | 2.06 | |
| 49 | 77.2 | $435.00 | 0.18 | |
| 50 | 77.0 | $0.0000 | — |