DeepSWE v1.1 evaluates coding agents on 113 original, long-horizon software-engineering tasks by grading committed patches in a clean, isolated environment with structured per-test reports.
Original tasks reduce answer leakage; behavioral verifiers accept alternative correct solutions and the long-horizon workload separates frontier coding systems.
Scores depend on the Mini-SWE-Agent harness and must preserve effort, costs, tokens, steps, attempts, and v1.1 grader identity. Repository and language coverage is concentrated in popular open-source projects, so transfer to proprietary, long-tail, localization, and refactoring work is uncertain.
Higher is better. Pass@1 across 113 tasks.
Max score: 100
Scores depend on the Mini-SWE-Agent harness and must preserve effort, costs, tokens, steps, attempts, and v1.1 grader identity.
No scores available for this benchmark yet.