A continuously updated software-engineering leaderboard using fresh GitHub tasks, a fixed minimal ReAct-style scaffold, standardized prompts, and five runs per model.
Tracks task dates and standardizes the scaffold and context policy. Reports pass@1, SEM, and pass@5 so model comparisons expose stochastic variance.
Automatically collected tasks are not guaranteed to be fully solvable or described well, which can depress resolve rates. Task windows and contamination differ by model, and the separately released V2 dataset is not proof that the rolling leaderboard reports V2 results.
Higher is better. Mean resolved rate (pass@1) across five runs; inspect SEM and pass@5 alongside it.
Max score: 100
Automatically collected tasks are not guaranteed to be fully solvable or described well, which can depress resolve rates.
| # | Model | Score |
|---|---|---|
| 1 | Anthropic: Claude Fable 5.1 source swe_rebench · Fable 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Fable 5 [high]__tools · record Fable 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 78.37837837837837,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2518308.0918918923,
"modelReleaseDate": "2026-06-09",
"scoreStandardError": 1.4130078505728048,
"cachedTokenPercentage": 94.9380118136434,
"modelReleaseTimestamp": 1780963200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 4.400018552252252,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 64.5 |
| 2 | Anthropic: Claude Fable 5 (batch) source swe_rebench · Fable 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Fable 5 [high]__tools · record Fable 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 78.37837837837837,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2518308.0918918923,
"modelReleaseDate": "2026-06-09",
"scoreStandardError": 1.4130078505728048,
"cachedTokenPercentage": 94.9380118136434,
"modelReleaseTimestamp": 1780963200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 4.400018552252252,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 64.5 |
| 3 | Grok 4.5 (High) source swe_rebench · Grok 4.5 [high] · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Grok 4.5 [high]__tools · record Grok 4.5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 77.47747747747748,
"developer": "xai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2429423.823423424,
"modelReleaseDate": "2026-07-08",
"scoreStandardError": 0.5975900523162895,
"cachedTokenPercentage": 92.59017890332878,
"modelReleaseTimestamp": 1783468800000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.4705187747747748,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 63.8 |
| 4 | Grok 4.5 (High) source swe_rebench · Grok 4.5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Grok 4.5 [high]__tools · record Grok 4.5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 77.47747747747748,
"developer": "xai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2429423.823423424,
"modelReleaseDate": "2026-07-08",
"scoreStandardError": 0.5975900523162895,
"cachedTokenPercentage": 92.59017890332878,
"modelReleaseTimestamp": 1783468800000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.4705187747747748,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 63.8 |
| 5 | Claude Opus 5 (Fast) source swe_rebench · Opus 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Opus 5 [high]__tools · record Opus 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4322143.001801802,
"modelReleaseDate": "2026-07-24",
"scoreStandardError": 1.35435102507864,
"cachedTokenPercentage": 95.70555353401815,
"modelReleaseTimestamp": 1784851200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 3.4657367509009007,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 63.4 |
| 6 | Claude Opus 5.5 source swe_rebench · Opus 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Opus 5 [high]__tools · record Opus 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4322143.001801802,
"modelReleaseDate": "2026-07-24",
"scoreStandardError": 1.35435102507864,
"cachedTokenPercentage": 95.70555353401815,
"modelReleaseTimestamp": 1784851200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 3.4657367509009007,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 63.4 |
| 7 | Claude Opus 5 (High) source swe_rebench · Opus 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Opus 5 [high]__tools · record Opus 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4322143.001801802,
"modelReleaseDate": "2026-07-24",
"scoreStandardError": 1.35435102507864,
"cachedTokenPercentage": 95.70555353401815,
"modelReleaseTimestamp": 1784851200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 3.4657367509009007,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 63.4 |
| 8 | GPT-5.5 (Xhigh) source swe_rebench · gpt-5.5-2026-04-23-xhigh · xhigh · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.5-2026-04-23-xhigh__tools · record gpt-5.5-2026-04-23-xhigh__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 70,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2120660.0327272727,
"modelReleaseDate": "2026-04-23",
"scoreStandardError": 0.9090909090909094,
"cachedTokenPercentage": 90.03088404558557,
"modelReleaseTimestamp": 1776902400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 2.245956852727273,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 62.7 |
| 9 | GLM-5.1 (Reasoning) source swe_rebench · GLM-5.1 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GLM-5.1__tools · record GLM-5.1__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 71.9298245614035,
"developer": "zhipu",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2227217.35093985,
"modelReleaseDate": "2026-04-07",
"scoreStandardError": 0.7393483709273188,
"cachedTokenPercentage": 92.31927667861885,
"modelReleaseTimestamp": 1775520000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.7587628127669174,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 62.7 |
| 10 | GPT-5.6 Sol (Medium) source swe_rebench · GPT-5.6 Sol [medium] · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GPT-5.6 Sol [medium]__tools · record GPT-5.6 Sol [medium]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 79.27927927927928,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 605339.9819819819,
"modelReleaseDate": "2026-06-26",
"scoreStandardError": 1.833062157601424,
"cachedTokenPercentage": 84.69228142556338,
"modelReleaseTimestamp": 1782432000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.8496136234234235,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 62.3 |
| 11 | GPT-5.6 Sol (Medium) source swe_rebench · GPT-5.6 Sol [medium] · medium · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GPT-5.6 Sol [medium]__tools · record GPT-5.6 Sol [medium]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 79.27927927927928,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 605339.9819819819,
"modelReleaseDate": "2026-06-26",
"scoreStandardError": 1.833062157601424,
"cachedTokenPercentage": 84.69228142556338,
"modelReleaseTimestamp": 1782432000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.8496136234234235,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 62.3 |
| 12 | Qwen3.5 397B A17B (Reasoning) source swe_rebench · Qwen3.5-397B-A17B · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Qwen3.5-397B-A17B__tools · record Qwen3.5-397B-A17B__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 71.9298245614035,
"developer": "alibaba",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2957222.26660401,
"modelReleaseDate": "2026-02-15",
"scoreStandardError": 1.6354943798653636,
"cachedTokenPercentage": 91.38288729053994,
"modelReleaseTimestamp": 1771113600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.1845025964473685,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 59.9 |
| 13 | Step 3.5 Flash source swe_rebench · Step-3.5-Flash · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Step-3.5-Flash__tools · record Step-3.5-Flash__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 71.9298245614035,
"developer": "stepfun",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 3745322.554511278,
"modelReleaseDate": "2026-02-01",
"scoreStandardError": 0.5404839044662804,
"cachedTokenPercentage": 81.88140770396407,
"modelReleaseTimestamp": 1769904000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.14399167301002508,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 59.6 |
| 14 | GPT-5.2 (Medium) source swe_rebench · gpt-5.2-2025-12-11-medium · medium · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-11-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.2-2025-12-11-medium__tools · record gpt-5.2-2025-12-11-medium__tools:1761955200000:1772323200000 View run sourceSource metadata{
"passAt5": 70,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 950013.0462660765,
"modelReleaseDate": "2025-12-11",
"scoreStandardError": 0.527707051151406,
"cachedTokenPercentage": 71.98812545693978,
"modelReleaseTimestamp": 1765411200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.676140301569186,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1761955200000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1761955200000
}
}Evidence observed Mar 1, 2026 | 59.3 |
| 15 | Qwen3.5 27B (Reasoning) source swe_rebench · Qwen3.5-27B · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Qwen3.5-27B__tools · record Qwen3.5-27B__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 64.91228070175438,
"developer": "alibaba",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2617752.270175439,
"modelReleaseDate": "2026-02-15",
"scoreStandardError": 0.7017543859649119,
"cachedTokenPercentage": 94.91762941099825,
"modelReleaseTimestamp": 1771113600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.5309000963157895,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 58.9 |
| 16 | GPT-5.5 (Medium) source swe_rebench · gpt-5.5-2026-04-23-medium · medium · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.5-2026-04-23-medium__tools · record gpt-5.5-2026-04-23-medium__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 70,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 708418.1072727272,
"modelReleaseDate": "2026-04-23",
"scoreStandardError": 0.7820295697311489,
"cachedTokenPercentage": 83.45910124001404,
"modelReleaseTimestamp": 1776902400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.9754196981818181,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 58.9 |
| 17 | GPT-5.3 Codex (Xhigh) source swe_rebench · gpt-5.3-codex-xhigh · xhigh · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.3-codex-xhigh__tools · record gpt-5.3-codex-xhigh__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 70.17543859649122,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1771027.652631579,
"modelReleaseDate": "2026-02-05",
"scoreStandardError": 0.8945648269461028,
"cachedTokenPercentage": 84.37114197273779,
"modelReleaseTimestamp": 1770249600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.8868153245614036,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 58.6 |
| 18 | GPT-5.3 Codex (Xhigh) source swe_rebench · gpt-5.3-codex · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.3-codex__tools · record gpt-5.3-codex__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 62.5,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 680434.392857143,
"modelReleaseDate": "2026-02-05",
"scoreStandardError": 1.3363062095621214,
"cachedTokenPercentage": 79.09955471129729,
"modelReleaseTimestamp": 1770249600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.40443676999999995,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 58.2 |
| 19 | GPT-4.5 (Preview) source swe_rebench · gpt-5.4-2026-03-05-medium · medium · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.4-2026-03-05-medium__tools · record gpt-5.4-2026-03-05-medium__tools:1769904000000:1778803200000 View run sourceSource metadata{
"passAt5": 70.65868263473054,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 814025.7628742515,
"modelReleaseDate": "2026-03-05",
"scoreStandardError": 0.8122550877994328,
"cachedTokenPercentage": 81.4876775099086,
"modelReleaseTimestamp": 1772668800000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6071112652694611,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
}
}Evidence observed May 15, 2026 | 57.6 |
| 20 | GPT-5.4 (Xhigh) source swe_rebench · gpt-5.4-2026-03-05-medium · medium · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.4-2026-03-05-medium__tools · record gpt-5.4-2026-03-05-medium__tools:1769904000000:1778803200000 View run sourceSource metadata{
"passAt5": 70.65868263473054,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 814025.7628742515,
"modelReleaseDate": "2026-03-05",
"scoreStandardError": 0.8122550877994328,
"cachedTokenPercentage": 81.4876775099086,
"modelReleaseTimestamp": 1772668800000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6071112652694611,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
}
}Evidence observed May 15, 2026 | 57.6 |
| 21 | GLM-5.2 (Max) source swe_rebench · GLM-5.2 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GLM-5.2 [high]__tools · record GLM-5.2 [high]__tools:1772323200000:1782864000000 View run sourceSource metadata{
"passAt5": 76.47058823529412,
"developer": "zhipu",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4080738.394570136,
"modelReleaseDate": "2026-05-16",
"scoreStandardError": 0.8585369213126834,
"cachedTokenPercentage": 89.59323708263648,
"modelReleaseTimestamp": 1778889600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.0782920420995477,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1772323200000
}
}Evidence observed Jul 1, 2026 | 57.0 |
| 22 | Claude Sonnet 5.5 source swe_rebench · Sonnet 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Sonnet 5 [high]__tools · record Sonnet 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4645616.863063063,
"modelReleaseDate": "2026-06-30",
"scoreStandardError": 0.9448728361893257,
"cachedTokenPercentage": 96.41145528118132,
"modelReleaseTimestamp": 1782777600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.4258611596396396,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 56.8 |
| 23 | Anthropic: Claude Sonnet 5 (batch) source swe_rebench · Sonnet 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Sonnet 5 [high]__tools · record Sonnet 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4645616.863063063,
"modelReleaseDate": "2026-06-30",
"scoreStandardError": 0.9448728361893257,
"cachedTokenPercentage": 96.41145528118132,
"modelReleaseTimestamp": 1782777600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.4258611596396396,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 56.8 |
| 24 | Claude Sonnet 5 (High) source swe_rebench · Sonnet 5 [high] · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-05-15:2026-07-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Sonnet 5 [high]__tools · record Sonnet 5 [high]__tools:1778803200000:1782864000000 View run sourceSource metadata{
"passAt5": 74.77477477477478,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4645616.863063063,
"modelReleaseDate": "2026-06-30",
"scoreStandardError": 0.9448728361893257,
"cachedTokenPercentage": 96.41145528118132,
"modelReleaseTimestamp": 1782777600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.4258611596396396,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
},
"publishedTaskRangeTimestamp": {
"to": 1782864000000,
"from": 1778803200000
}
}Evidence observed Jul 1, 2026 | 56.8 |
| 25 | GPT-5.2 (Xhigh) source swe_rebench · gpt-5.2-2025-12-11-xhigh · xhigh · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-12-01:2026-02-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.2-2025-12-11-xhigh__tools · record gpt-5.2-2025-12-11-xhigh__tools:1764547200000:1769904000000 View run sourceSource metadata{
"passAt5": 64.58333333333334,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1909938.5220175437,
"modelReleaseDate": "2025-12-11",
"scoreStandardError": 0.3045917639386849,
"cachedTokenPercentage": 74.35348070732941,
"modelReleaseTimestamp": 1765411200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.371884737654605,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1764547200000
},
"publishedTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1764547200000
}
}Evidence observed Feb 1, 2026 | 56.6 |
| 26 | Claude Opus 4.8 (Max) source swe_rebench · Claude Opus 4.8-xhigh · xhigh · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Opus 4.8-xhigh__tools · record Claude Opus 4.8-xhigh__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 67.27272727272727,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2479387.470909091,
"modelReleaseDate": "2026-05-28",
"scoreStandardError": 1.199173268933903,
"cachedTokenPercentage": 95.27838279131717,
"modelReleaseTimestamp": 1779926400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 2.0202793086363635,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 56.5 |
| 27 | Gemini 3.1 Pro Preview source swe_rebench · Gemini 3.1 Pro Preview · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Gemini 3.1 Pro Preview__tools · record Gemini 3.1 Pro Preview__tools:1769904000000:1778803200000 View run sourceSource metadata{
"passAt5": 69.46107784431138,
"developer": "google",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1502695.5944015582,
"modelReleaseDate": "2026-02-19",
"scoreStandardError": 0.9239067816830353,
"cachedTokenPercentage": 80.60600910081673,
"modelReleaseTimestamp": 1771459200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.7208578944116585,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
}
}Evidence observed May 15, 2026 | 54.9 |
| 28 | Claude Sonnet 4.6 (Non-reasoning, High) source swe_rebench · Claude Sonnet 4.6 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Sonnet 4.6__tools · record Claude Sonnet 4.6__tools:1769904000000:1778803200000 View run sourceSource metadata{
"passAt5": 65.86826347305389,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2454257.6670658686,
"modelReleaseDate": "2026-02-17",
"scoreStandardError": 0.18935794372265716,
"cachedTokenPercentage": 95.4617277923743,
"modelReleaseTimestamp": 1771286400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.1958073514371255,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1769904000000
}
}Evidence observed May 15, 2026 | 54.5 |
| 29 | OpenAI: GPT-5.1-Codex-Max source swe_rebench · gpt-5.1-codex-max · max · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-11-01:2026-02-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.1-codex-max__tools · record gpt-5.1-codex-max__tools:1761955200000:1769904000000 View run sourceSource metadata{
"passAt5": 66.43356643356644,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1308803.2846055352,
"modelReleaseDate": "2025-11-19",
"scoreStandardError": 1.597583179363583,
"cachedTokenPercentage": 72.71792333880192,
"modelReleaseTimestamp": 1763510400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.649101685235891,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1761955200000
},
"publishedTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1761955200000
}
}Evidence observed Feb 1, 2026 | 54.5 |
| 30 | Qwen3.5 35B A3B (Reasoning) source swe_rebench · Qwen3.5-35B-A3B · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Qwen3.5-35B-A3B__tools · record Qwen3.5-35B-A3B__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 63.1578947368421,
"developer": "alibaba",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 4753718.8912280705,
"modelReleaseDate": "2026-02-15",
"scoreStandardError": 2.5180175604223383,
"cachedTokenPercentage": 94.34453785997876,
"modelReleaseTimestamp": 1771113600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.8062736944736842,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 53.7 |
| 31 | GLM-5 (Reasoning) source swe_rebench · GLM-5 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GLM-5__tools · record GLM-5__tools:1767225600000:1772323200000 View run sourceSource metadata{
"passAt5": 60.952380952380956,
"developer": "zhipu",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1856559.6152380952,
"modelReleaseDate": "2026-02-11",
"scoreStandardError": 0.9035079029052521,
"cachedTokenPercentage": 81.18903430513568,
"modelReleaseTimestamp": 1770768000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6191998198095238,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
}
}Evidence observed Mar 1, 2026 | 53.3 |
| 32 | Claude Opus 4.6 (Non-reasoning, High) source swe_rebench · Claude Opus 4.6-high · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Opus 4.6-high__tools · record Claude Opus 4.6-high__tools:1767225600000:1778803200000 View run sourceSource metadata{
"passAt5": 62.7906976744186,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1509615.6027906977,
"modelReleaseDate": "2026-02-05",
"scoreStandardError": 0.5992604058941919,
"cachedTokenPercentage": 93.58694505375718,
"modelReleaseTimestamp": 1770249600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.2870668527906977,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1767225600000
}
}Evidence observed May 15, 2026 | 53.3 |
| 33 | Claude Opus 4.5 (Reasoning) source swe_rebench · Claude Opus 4.5 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-10-01:2026-02-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Opus 4.5__tools · record Claude Opus 4.5__tools:1759276800000:1769904000000 View run sourceSource metadata{
"passAt5": 65.4639175257732,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1358096.175257732,
"modelReleaseDate": "2025-11-24",
"scoreStandardError": 0.4312680549144704,
"cachedTokenPercentage": 95.173436766734,
"modelReleaseTimestamp": 1763942400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.1468206239690724,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1759276800000
},
"publishedTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1759276800000
}
}Evidence observed Feb 1, 2026 | 53.1 |
| 34 | Claude Opus 4.7 source swe_rebench · Claude Opus 4.7-high · high · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Opus 4.7-high__tools · record Claude Opus 4.7-high__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 66.36363636363637,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1526135.3290909089,
"modelReleaseDate": "2026-04-16",
"scoreStandardError": 1.4545454545454541,
"cachedTokenPercentage": 94.18072169671689,
"modelReleaseTimestamp": 1776297600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.3181382954545455,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 53.1 |
| 35 | Gemini 3 Flash Preview (Reasoning) source swe_rebench · Gemini 3 Flash Preview · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-12-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Gemini 3 Flash Preview__tools · record Gemini 3 Flash Preview__tools:1764547200000:1772323200000 View run sourceSource metadata{
"passAt5": 65.359477124183,
"developer": "google",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2011252.2615411077,
"modelReleaseDate": "2025-12-17",
"scoreStandardError": 0.4168980195749748,
"cachedTokenPercentage": 75.9805341681911,
"modelReleaseTimestamp": 1765929600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.2892457823095115,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
}
}Evidence observed Mar 1, 2026 | 53.0 |
| 36 | Claude 4.5 Sonnet (Reasoning) source swe_rebench · Claude Sonnet 4.5 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-09-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Claude Sonnet 4.5__tools · record Claude Sonnet 4.5__tools:1756684800000:1772323200000 View run sourceSource metadata{
"passAt5": 64.54849498327759,
"developer": "anthropic",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1995406.6802675587,
"modelReleaseDate": "2025-09-29",
"scoreStandardError": 0.8287298791547733,
"cachedTokenPercentage": null,
"modelReleaseTimestamp": 1759104000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.9791713679598661,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1756684800000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1756684800000
}
}Evidence observed Mar 1, 2026 | 52.4 |
| 37 | MiniMax-M2.7 source swe_rebench · MiniMax M2.7 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-02-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider MiniMax M2.7__tools · record MiniMax M2.7__tools:1769904000000:1772323200000 View run sourceSource metadata{
"passAt5": 61.40350877192983,
"developer": "minimax",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2978534.3031954886,
"modelReleaseDate": "2026-03-18",
"scoreStandardError": 0.9416440680638873,
"cachedTokenPercentage": 92.21263735003976,
"modelReleaseTimestamp": 1773792000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.23654764124154135,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1769904000000
}
}Evidence observed Mar 1, 2026 | 51.9 |
| 38 | GPT-5.2 Codex (Xhigh) source swe_rebench · gpt-5.2-codex · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.2-codex__tools · record gpt-5.2-codex__tools:1767225600000:1772323200000 View run sourceSource metadata{
"passAt5": 59.61538461538461,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 661953.5403846154,
"modelReleaseDate": "2025-12-18",
"scoreStandardError": 1.2388556468004936,
"cachedTokenPercentage": 70.83085152655448,
"modelReleaseTimestamp": 1766016000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.4398054520192307,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
}
}Evidence observed Mar 1, 2026 | 51.3 |
| 39 | GLM-5.2 (Max) source swe_rebench · GLM-5.2 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GLM-5.2__tools · record GLM-5.2__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 71.81818181818181,
"developer": "zhipu",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2623455.730909091,
"modelReleaseDate": "2026-05-16",
"scoreStandardError": 1.1281521496355318,
"cachedTokenPercentage": 87.02682379005734,
"modelReleaseTimestamp": 1778889600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.7507865763818182,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 51.1 |
| 40 | Gemini 3 Pro Preview (Low) source swe_rebench · Gemini 3 Pro Preview · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-10-01:2026-02-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Gemini 3 Pro Preview__tools · record Gemini 3 Pro Preview__tools:1759276800000:1769904000000 View run sourceSource metadata{
"passAt5": 62.88659793814433,
"developer": "google",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1265245.1555312215,
"modelReleaseDate": "2025-11-19",
"scoreStandardError": 1.2239108397606742,
"cachedTokenPercentage": 81.85933795481792,
"modelReleaseTimestamp": 1763510400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6534790917333474,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1759276800000
},
"publishedTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1759276800000
}
}Evidence observed Feb 1, 2026 | 50.7 |
| 41 | GLM-5.1 (Reasoning) source swe_rebench · GLM-5.1 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider GLM-5.1__tools · record GLM-5.1__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 65.45454545454545,
"developer": "zhipu",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2664000.992727273,
"modelReleaseDate": "2026-04-07",
"scoreStandardError": 0.9270944570168703,
"cachedTokenPercentage": 91.78128852041365,
"modelReleaseTimestamp": 1775520000000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.9428887529454546,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 50.7 |
| 42 | Gemini 3.5 Flash (High) source swe_rebench · Gemini 3.5 Flash · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Gemini 3.5 Flash__tools · record Gemini 3.5 Flash__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 61.81818181818181,
"developer": "google",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1848593.0709090908,
"modelReleaseDate": "2026-05-19",
"scoreStandardError": 0.9791208740244552,
"cachedTokenPercentage": 75.66134999998506,
"modelReleaseTimestamp": 1779148800000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.7666600170000002,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 49.5 |
| 43 | Kimi K2.5 (Reasoning) source swe_rebench · Kimi K2.5 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Kimi K2.5__tools · record Kimi K2.5__tools:1767225600000:1772323200000 View run sourceSource metadata{
"passAt5": 60.952380952380956,
"developer": "moonshot",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1361950.016043956,
"modelReleaseDate": "2026-01-26",
"scoreStandardError": 0.7979212617639928,
"cachedTokenPercentage": 92.2213727004562,
"modelReleaseTimestamp": 1769385600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.19543760866117216,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
}
}Evidence observed Mar 1, 2026 | 49.0 |
| 44 | DeepSeek V3.2 (Non-reasoning) source swe_rebench · DeepSeek-V3.2 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-11-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider DeepSeek-V3.2__tools · record DeepSeek-V3.2__tools:1761955200000:1772323200000 View run sourceSource metadata{
"passAt5": 65,
"developer": "deepseek",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2349749.462790721,
"modelReleaseDate": "2025-12-01",
"scoreStandardError": 0.763306626834098,
"cachedTokenPercentage": null,
"modelReleaseTimestamp": 1764547200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.44236133086516966,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1761955200000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1761955200000
}
}Evidence observed Mar 1, 2026 | 48.9 |
| 45 | GPT-5.1 Codex (High) source swe_rebench · gpt-5.1-codex · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-12-01:2026-02-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider gpt-5.1-codex__tools · record gpt-5.1-codex__tools:1764547200000:1769904000000 View run sourceSource metadata{
"passAt5": 57.291666666666664,
"developer": "openai",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1684056.2770833336,
"modelReleaseDate": "2025-11-12",
"scoreStandardError": 0.8960755486502733,
"cachedTokenPercentage": 81.83799029410963,
"modelReleaseTimestamp": 1762905600000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6300463354166667,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1764547200000
},
"publishedTaskRangeTimestamp": {
"to": 1769904000000,
"from": 1764547200000
}
}Evidence observed Feb 1, 2026 | 48.8 |
| 46 | Kimi K2 Thinking source swe_rebench · Kimi K2 Thinking · thinking · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-12-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Kimi K2 Thinking__tools · record Kimi K2 Thinking__tools:1764547200000:1772323200000 View run sourceSource metadata{
"passAt5": 64.05228758169935,
"developer": "moonshot",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2620382.7760665626,
"modelReleaseDate": "2025-11-06",
"scoreStandardError": 1.2345501100854865,
"cachedTokenPercentage": 93.80388292409071,
"modelReleaseTimestamp": 1762387200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.5121233394787587,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
}
}Evidence observed Mar 1, 2026 | 47.8 |
| 47 | Kimi K2 Thinking source swe_rebench · Kimi K2 Thinking · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2025-12-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Kimi K2 Thinking__tools · record Kimi K2 Thinking__tools:1764547200000:1772323200000 View run sourceSource metadata{
"passAt5": 64.05228758169935,
"developer": "moonshot",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2620382.7760665626,
"modelReleaseDate": "2025-11-06",
"scoreStandardError": 1.2345501100854865,
"cachedTokenPercentage": 93.80388292409071,
"modelReleaseTimestamp": 1762387200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.5121233394787587,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1764547200000
}
}Evidence observed Mar 1, 2026 | 47.8 |
| 48 | Qwen3 Coder Next source swe_rebench · Qwen3-Coder-Next · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Qwen3-Coder-Next__tools · record Qwen3-Coder-Next__tools:1767225600000:1772323200000 View run sourceSource metadata{
"passAt5": 63.8095238095238,
"developer": "alibaba",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 5477088.4599816855,
"modelReleaseDate": "2026-02-02",
"scoreStandardError": 0.8884973862435783,
"cachedTokenPercentage": 98.05739917506962,
"modelReleaseTimestamp": 1769990400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 1.1246296911153846,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
}
}Evidence observed Mar 1, 2026 | 47.8 |
| 49 | MiniMax-M2.5 source swe_rebench · MiniMax M2.5 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-01-01:2026-03-01 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider MiniMax M2.5__tools · record MiniMax M2.5__tools:1767225600000:1772323200000 View run sourceSource metadata{
"passAt5": 60,
"developer": "minimax",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 1897675.4177655678,
"modelReleaseDate": "2026-02-12",
"scoreStandardError": 1.3664783491887242,
"cachedTokenPercentage": 89.96401144503699,
"modelReleaseTimestamp": 1770854400000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.11185288953434067,
"potentialContamination": true,
"aggregateTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
},
"publishedTaskRangeTimestamp": {
"to": 1772323200000,
"from": 1767225600000
}
}Evidence observed Mar 1, 2026 | 47.7 |
| 50 | Kimi K2.6 (Reasoning) source swe_rebench · Kimi K2.6 · swe-rebench-react@tools · benchmark continuous-leaderboard · split tasks-created:2026-03-01:2026-05-15 · 5 attempts · pass@1 · swe-rebench:standardized:128k-or-model-limit · provider Kimi K2.6__tools · record Kimi K2.6__tools:1772323200000:1778803200000 View run sourceSource metadata{
"passAt5": 64.54545454545455,
"developer": "moonshot",
"instanceType": "model",
"contextPolicy": "128k unless the model supports less",
"attemptsPerTask": 5,
"totalTokenUsage": 2466976.5963636367,
"modelReleaseDate": "2026-03-01",
"scoreStandardError": 1.2662171161076474,
"cachedTokenPercentage": 90.35377561544833,
"modelReleaseTimestamp": 1772323200000,
"standardContextTokens": 128000,
"averageInstanceCostUsd": 0.6078855246545455,
"potentialContamination": false,
"aggregateTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
},
"publishedTaskRangeTimestamp": {
"to": 1778803200000,
"from": 1772323200000
}
}Evidence observed May 15, 2026 | 46.5 |