SWE-Bench Pro evaluates coding agents on 731 long-horizon tasks from public copyleft repositories using executable tests and resolve rate.
Covers substantial changes across professional repositories; publishes trajectories and separates capped from uncapped agent configurations.
OpenAI estimates that roughly 30% of SWE-Bench Pro tasks are broken because of strict tests, underspecified or misleading prompts, and low test coverage. Resolve rate is an agent-system result, so preserve harness, effort, cost and turn caps, split, and observation date.
Higher is better. Percentage of tasks resolved correctly.
OpenAI estimates that roughly 30% of SWE-Bench Pro tasks are broken because of strict tests, underspecified or misleading prompts, and low test coverage.
| # | Model | Score |
|---|---|---|
| 1 | 61.5 | |
| 2 | 59.1 | |
| 3 | 59.1 | |
| 4 | 55.0 | |
| 5 | 51.9 | |
| 6 | 46.1 | |
| 7 | 45.9 | |
| 8 | 45.9 | |
| 9 | 43.6 |