A 100-task benchmark (50 public and 50 private) of feature, bug, performance, and migration work drawn from substantial real pull requests and evaluated with behavioral and code-quality criteria.
Uses naturally underspecified senior-engineer tasks across diverse stacks and long trajectories. Adds adaptive behavioral validation and codebase-practice criteria beyond test passing.
The tasteful metric is judge-sensitive, so preserve its judge and rubric version separately from basic verifier pass@1. The public and private halves have different exposure risk, and scores remain Mini-SWE-Agent system results whose effort, run set, costs, tokens, and steps must be preserved.
Higher is better. AgMoDB stores tasteful pass@1 as the primary score and preserves basic pass@1 as source metadata.
Max score: 100
Showing the default model and evaluation cohort.
google/gemini-3.5-flash · senior_swe_bench · 2026.06 · official-100-task · mini-swe-agent · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · token budget none · cost budget none · action budget none · per task
The tasteful metric is judge-sensitive, so preserve its judge and rubric version separately from basic verifier pass@1.
| # | Model | Score |
|---|---|---|
| 1 | Claude Fable 5 (Max, Opus 4.8 Fallback) source senior_swe_bench · Claude Fable 5 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-fable-5 · record fable-5-mini-swe · run fbe85de7c1b2b22d1cb6b0e746b610642eeeafccbd19cb4693b6101a8353e9fa View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-fable-5",
"basicPassAt1": 46.51162790697674,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 29.069767441860467,
"agentDescription": "Claude Fable 5 via mini-swe-agent at adaptive-thinking effort=max (headline)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 43,
"averageAgentSteps": 158.50588235294117,
"averageOutputTokens": 290182.9529411765,
"averageTotalCostUsd": 29.373847769411764,
"averageOutputCostUsd": 14.509147647058823
}Evidence observed Jul 4, 2026 | 29.1 |
| 2 | Claude Opus 5 (Xhigh) source senior_swe_bench · Claude Opus 5 (xhigh) · xhigh · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-horchata-eap · record horchata-xhigh-mini-swe · run 2b77b215194529178aee78d7f705aaaf2b675a96937a6345c6d8dddab183b2a0 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-horchata-eap",
"basicPassAt1": 51.76470588235295,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 28.235294117647058,
"agentDescription": "Claude Opus 5 (horchata EAP) via mini-swe-agent at adaptive-thinking effort=xhigh",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 49,
"averageAgentSteps": 158.04705882352943,
"averageOutputTokens": 93165.14117647058,
"averageTotalCostUsd": 11.688398689411764,
"averageOutputCostUsd": 2.3291285294117645
}Evidence observed Jul 24, 2026 | 28.2 |
| 3 | Claude Sonnet 5 (Max) source senior_swe_bench · Claude Sonnet 5 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-sonnet-5 · record sonnet-5-bed-mini-swe · run 7f13b0ad9a70cf6c02d279345f50812ad030149bea9991d3eac8101cbf78ef43 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-sonnet-5",
"basicPassAt1": 50.53763440860215,
"basicPassAt3": 59.13978494623656,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 27.956989247311824,
"tastefulPassAt3": 37.634408602150536,
"agentDescription": "TEMP: Sonnet 5 @bedrock half of the provider split (effort=max)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 49,
"averageAgentSteps": 318.4623655913978,
"averageOutputTokens": 180719.59139784946,
"averageTotalCostUsd": 23.426825863225808,
"averageOutputCostUsd": 2.7107938709677417,
"basicAllThreePassRate": 34.40860215053764,
"tastefulAllThreePassRate": 13.978494623655912
}Evidence observed Jul 22, 2026 | 28.0 |
| 4 | Grok 4.6 (High) source senior_swe_bench · Grok 4.6 (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/grok-4.6 · record grok-4-6-high-mini-swe · run e110d5b781a85ef95ec9f62e3e7403f974f2a8b2e65c04e7740f70a18d58dcea View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:grok-4.6",
"basicPassAt1": 51.578947368421055,
"basicPassAt3": 65.26315789473685,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 26.31578947368421,
"tastefulPassAt3": 38.94736842105263,
"agentDescription": "xAI Grok 4.6 via mini-swe-agent, direct api.x.ai, reasoning_effort=high",
"privateTaskCount": 50,
"upstreamProvider": "xai",
"upstreamRunCount": 50,
"averageAgentSteps": 124.57894736842105,
"averageOutputTokens": 23923.547368421052,
"averageTotalCostUsd": 0,
"averageOutputCostUsd": 0,
"basicAllThreePassRate": 35.78947368421053,
"tastefulAllThreePassRate": 16.842105263157894
}Evidence observed Aug 13, 2026 | 26.3 |
| 5 | Claude Opus 4.8 (Max) source senior_swe_bench · Claude Opus 4.8 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-opus-4-8 · record opus-48-mini-swe · run 4ee23dca8f25cd97660a6a5e6781ba4c8a820780bc8018dd43fcf22a96c06b11 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-opus-4-8",
"basicPassAt1": 39.77272727272727,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 25,
"agentDescription": "Claude Opus 4.8 via mini-swe-agent harness (Portkey→Bedrock)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 80,
"averageAgentSteps": 132.36363636363637,
"averageOutputTokens": 115169.26136363637,
"averageTotalCostUsd": 8.03432186590909,
"averageOutputCostUsd": 2.879231534090909
}Evidence observed Jun 30, 2026 | 25.0 |
| 6 | GPT-5.6 Sol (Xhigh) source senior_swe_bench · GPT-5.6 Sol (xhigh) · xhigh · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/gpt-5.6-sol · record gpt-5-6-mini-swe · run 8f945404bd658690c5cc5c3cebb963696d2a004fefd885318a759e3324beacdf View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gpt-5.6-sol",
"basicPassAt1": 54.65116279069767,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 24.418604651162788,
"agentDescription": "OpenAI GPT-5.6 (gpt-5.6-sol) via mini-swe-agent harness (Portkey→OpenAI)",
"privateTaskCount": 50,
"upstreamProvider": "openai",
"upstreamRunCount": 49,
"averageAgentSteps": 48.54022988505747,
"averageOutputTokens": 31106.379310344826,
"averageTotalCostUsd": 3.1808865379310345,
"averageOutputCostUsd": 0.9331913793103448
}Evidence observed Jul 8, 2026 | 24.4 |
| 7 | MiniMax-M3 source senior_swe_bench · MiniMax M3 (adaptive) · adaptive · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/minimax/minimax-m3 · record minimax-m3-mini-swe · run 40e12d37c11a5700209734f0cd22d3e42aae26b7d6ae1c19d13dd547769acb3b View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:minimax-m3",
"basicPassAt1": 41.05263157894737,
"basicPassAt3": 50.526315789473685,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 21.052631578947366,
"tastefulPassAt3": 28.421052631578945,
"agentDescription": "MiniMax M3 via mini-swe (Portkey->OpenRouter), adaptive thinking",
"privateTaskCount": 50,
"upstreamProvider": "minimax",
"upstreamRunCount": 50,
"averageAgentSteps": 335.33684210526314,
"averageOutputTokens": 75786.57894736843,
"averageTotalCostUsd": 1.2684179555368422,
"averageOutputCostUsd": 0.09094389473684211,
"basicAllThreePassRate": 21.052631578947366,
"tastefulAllThreePassRate": 8.421052631578947
}Evidence observed Jul 30, 2026 | 21.1 |
| 8 | Kimi K3 source senior_swe_bench · Kimi K3 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider moonshotai/kimi-k3 · record kimi-k3-mini-swe · run 45efa2b5e98d0e1b72b1d164d138bae8477c2fe17aa4a831e8999424bc24459f View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:kimi-k3",
"basicPassAt1": 40.42553191489361,
"basicPassAt3": 57.446808510638306,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 20.212765957446805,
"tastefulPassAt3": 35.1063829787234,
"agentDescription": "Moonshot Kimi K3 via mini-swe-agent harness (Portkey→OpenRouter); reasoning mandatory (max only)",
"privateTaskCount": 50,
"upstreamProvider": "kimi",
"upstreamRunCount": 50,
"averageAgentSteps": 223.80851063829786,
"averageOutputTokens": 54818.574468085106,
"averageTotalCostUsd": 7.024542418723404,
"averageOutputCostUsd": 0.8222786170212766,
"basicAllThreePassRate": 22.340425531914892,
"tastefulAllThreePassRate": 6.382978723404255
}Evidence observed Jul 29, 2026 | 20.2 |
| 9 | Claude Opus 4.7 (Max) source senior_swe_bench · Claude Opus 4.7 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-opus-4-7 · record opus-mini-swe · run aa0d49c0880d35d45f85fa7ee0c9f7bc0bc261fd1803632a0b2f0e96eff1b538 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-opus-4-7",
"basicPassAt1": 46.808510638297875,
"basicPassAt3": 56.38297872340425,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 18.085106382978726,
"tastefulPassAt3": 34.04255319148936,
"agentDescription": "Claude Opus 4.7 via mini-swe-agent harness (Portkey→Bedrock)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 50,
"averageAgentSteps": 284.63829787234044,
"averageOutputTokens": 100937.73404255319,
"averageTotalCostUsd": 21.950259461702124,
"averageOutputCostUsd": 2.5234433510638294,
"basicAllThreePassRate": 31.914893617021278,
"tastefulAllThreePassRate": 9.574468085106384
}Evidence observed Jul 22, 2026 | 18.1 |
| 10 | GLM-5.2 (Max) source senior_swe_bench · GLM-5.2 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider z-ai/glm-5.2 · record glm-5-2-mini-swe · run f724b2c25f0cd838b748ad20043c4035202286faf83467ccef519c27a34218b8 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:glm-5.2",
"basicPassAt1": 35.78947368421053,
"basicPassAt3": 49.473684210526315,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 17.894736842105264,
"tastefulPassAt3": 30.526315789473685,
"agentDescription": "Z.ai GLM-5.2 via mini-swe-agent harness (Portkey→OpenRouter)",
"privateTaskCount": 50,
"upstreamProvider": "zai",
"upstreamRunCount": 50,
"averageAgentSteps": 193.62105263157895,
"averageOutputTokens": 66230.33684210526,
"averageTotalCostUsd": 0.6412509064421053,
"averageOutputCostUsd": 0.08742404463157895,
"basicAllThreePassRate": 23.157894736842106,
"tastefulAllThreePassRate": 6.315789473684211
}Evidence observed Jul 28, 2026 | 17.9 |
| 11 | Claude Sonnet 5 (Max) source senior_swe_bench · Claude Sonnet 5 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-sonnet-5 · record sonnet-5-mini-swe · run 223953bb4638ed366183df51d3cce69709ef29fe264a65921e7ed96e4db8a307 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-sonnet-5",
"basicPassAt1": 39.53488372093023,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 17.441860465116278,
"agentDescription": "Claude Sonnet 5 via mini-swe-agent at adaptive-thinking effort=max (headline)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 48,
"averageAgentSteps": 333.1190476190476,
"averageOutputTokens": 445451.7619047619,
"averageTotalCostUsd": 21.683269077142857,
"averageOutputCostUsd": 6.681776428571428
}Evidence observed Jul 4, 2026 | 17.4 |
| 12 | Grok 4.5 (High) source senior_swe_bench · Grok 4.5 (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/grok-4.5 · record grok-4-5-mini-swe · run 1b426ea2b95513900b31967c6460a77b395f178aced1c84f631044114146fd16 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:grok-4.5",
"basicPassAt1": 49.42528735632184,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 17.24137931034483,
"agentDescription": "xAI Grok 4.5 via mini-swe-agent harness (Portkey→xAI)",
"privateTaskCount": 50,
"upstreamProvider": "xai",
"upstreamRunCount": 50,
"averageAgentSteps": 79.86363636363636,
"averageOutputTokens": 22893.670454545456,
"averageTotalCostUsd": 1.2253361936363636,
"averageOutputCostUsd": 0.13736202272727271
}Evidence observed Jul 9, 2026 | 17.2 |
| 13 | GPT-5.4 (Xhigh) source senior_swe_bench · GPT-5.4 (xhigh) · xhigh · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/gpt-5.4 · record gpt-5-4-mini-swe · run c243aeb40c0c89716ec5a6e2056652f84b289975286619d6e25927d25cbb69c6 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gpt-5.4",
"basicPassAt1": 41.05263157894737,
"basicPassAt3": 54.736842105263165,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 16.842105263157894,
"tastefulPassAt3": 27.368421052631582,
"agentDescription": "OpenAI GPT-5.4 via mini-swe-agent harness (Portkey→OpenAI)",
"privateTaskCount": 50,
"upstreamProvider": "openai",
"upstreamRunCount": 50,
"averageAgentSteps": 86.26315789473684,
"averageOutputTokens": 53854.221052631576,
"averageTotalCostUsd": 3.071602650526316,
"averageOutputCostUsd": 0.8078133157894736,
"basicAllThreePassRate": 24.210526315789473,
"tastefulAllThreePassRate": 6.315789473684211
}Evidence observed Jul 22, 2026 | 16.8 |
| 14 | GPT-5.5 (Xhigh) source senior_swe_bench · GPT-5.5 (xhigh) · xhigh · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/gpt-5.5 · record gpt-5-5-mini-swe · run a165e24cf1b9c072a876b82730459f2c240ba89568b769933669b4d2303dea36 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gpt-5.5",
"basicPassAt1": 56.81818181818182,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 15.909090909090908,
"agentDescription": "OpenAI GPT-5.5 via mini-swe-agent harness (Portkey→OpenAI)",
"privateTaskCount": 50,
"upstreamProvider": "openai",
"upstreamRunCount": 78,
"averageAgentSteps": 88.32954545454545,
"averageOutputTokens": 35910.045454545456,
"averageTotalCostUsd": 4.934496786363637,
"averageOutputCostUsd": 1.0773013636363638
}Evidence observed Jun 30, 2026 | 15.9 |
| 15 | Gemini 3.7 Flash (High) source senior_swe_bench · Gemini 3.7 Flash (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider google/gemini-3.7-flash · record gemini-3-7-flash-mini-swe · run 5bcc3d13da0a1a8d2b6b1bdc2b03eeae0d4958b29ec6e07e760d18767a104e2e View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gemini-3.7-flash",
"basicPassAt1": 44.21052631578947,
"basicPassAt3": 61.05263157894737,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 14.736842105263156,
"tastefulPassAt3": 24.210526315789473,
"agentDescription": "Google Gemini 3.7 Flash via mini-swe-agent harness (Portkey→Google)",
"privateTaskCount": 50,
"upstreamProvider": "google",
"upstreamRunCount": 50,
"averageAgentSteps": 254.49473684210525,
"averageOutputTokens": 47266.17894736842,
"averageTotalCostUsd": 2.7541014963157897,
"averageOutputCostUsd": 0.17724817105263158,
"basicAllThreePassRate": 31.57894736842105,
"tastefulAllThreePassRate": 6.315789473684211
}Evidence observed Aug 22, 2026 | 14.7 |
| 16 | MiniMax-M3 source senior_swe_bench · MiniMax M3 (adaptive) · adaptive · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/minimax/minimax-m3 · record minimax-m3-mini-swe · run ef8d583a4b3f6b6e11e0d4f03933e933f5d1e6e0bce960327f709628d888eee3 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:minimax-m3",
"basicPassAt1": 35.63218390804598,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 13.793103448275861,
"agentDescription": "MiniMax M3 via mini-swe (Portkey->OpenRouter), adaptive thinking",
"privateTaskCount": 50,
"upstreamProvider": "minimax",
"upstreamRunCount": 50,
"averageAgentSteps": 369.57954545454544,
"averageOutputTokens": 85256.39772727272,
"averageTotalCostUsd": 1.513936438090909,
"averageOutputCostUsd": 0.10230767727272727
}Evidence observed Jul 22, 2026 | 13.8 |
| 17 | Claude Opus 4.7 source senior_swe_bench · Claude Opus 4.7 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-opus-4-7 · record opus-mini-swe · run 1d69314975a9519f64c507748d04762ded7853ebf34982811a94ac3a27ba2f9e View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-opus-4-7",
"basicPassAt1": 39.08045977011494,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 13.793103448275861,
"agentDescription": "Claude Opus 4.7 via mini-swe-agent harness (Portkey→Bedrock)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 50,
"averageAgentSteps": 157.95402298850576,
"averageOutputTokens": 97643.86206896552,
"averageTotalCostUsd": 8.871201140229886,
"averageOutputCostUsd": 2.4410965517241383
}Evidence observed Jun 30, 2026 | 13.8 |
| 18 | Claude Opus 4.7 (Max) source senior_swe_bench · Claude Opus 4.7 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-opus-4-7 · record opus-mini-swe · run 1d69314975a9519f64c507748d04762ded7853ebf34982811a94ac3a27ba2f9e View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-opus-4-7",
"basicPassAt1": 39.08045977011494,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 13.793103448275861,
"agentDescription": "Claude Opus 4.7 via mini-swe-agent harness (Portkey→Bedrock)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 50,
"averageAgentSteps": 157.95402298850576,
"averageOutputTokens": 97643.86206896552,
"averageTotalCostUsd": 8.871201140229886,
"averageOutputCostUsd": 2.4410965517241383
}Evidence observed Jun 30, 2026 | 13.8 |
| 19 | GPT-5.4 (Xhigh) source senior_swe_bench · GPT-5.4 (xhigh) · xhigh · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/gpt-5.4 · record gpt-5-4-mini-swe · run 90509a67fed6a6131978e82ddf4b66f2848116a98bd228326ce0482485a2feb3 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gpt-5.4",
"basicPassAt1": 48.86363636363637,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 13.636363636363635,
"agentDescription": "OpenAI GPT-5.4 via mini-swe-agent harness (Portkey→OpenAI)",
"privateTaskCount": 50,
"upstreamProvider": "openai",
"upstreamRunCount": 50,
"averageAgentSteps": 81.7840909090909,
"averageOutputTokens": 50491.568181818184,
"averageTotalCostUsd": 2.9705425000000005,
"averageOutputCostUsd": 0.7573735227272728
}Evidence observed Jun 30, 2026 | 13.6 |
| 20 | GLM-5.2 (Max) source senior_swe_bench · GLM-5.2 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider z-ai/glm-5.2 · record glm-5-2-mini-swe · run 89c8d46b4c85d80818b46eb0e5c5121720590d74c6a1ce433c3386872ab13a52 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:glm-5.2",
"basicPassAt1": 32.142857142857146,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 13.095238095238097,
"agentDescription": "Z.ai GLM-5.2 via mini-swe-agent harness (Portkey→OpenRouter)",
"privateTaskCount": 50,
"upstreamProvider": "zai",
"upstreamRunCount": 50,
"averageAgentSteps": 213.5568181818182,
"averageOutputTokens": 65991.90909090909,
"averageTotalCostUsd": 0.7522987828909091,
"averageOutputCostUsd": 0.08710932
}Evidence observed Jul 1, 2026 | 13.1 |
| 21 | Kimi K3 source senior_swe_bench · Kimi K3 (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider moonshotai/kimi-k3 · record kimi-k3-mini-swe · run 132d565e2dcf4bc4526769bf487eba3743a0419460f03194909391f643ad5312 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:kimi-k3",
"basicPassAt1": 35.294117647058826,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 12.941176470588237,
"agentDescription": "Moonshot Kimi K3 via mini-swe-agent harness (Portkey→OpenRouter); reasoning mandatory (max only)",
"privateTaskCount": 50,
"upstreamProvider": "kimi",
"upstreamRunCount": 50,
"averageAgentSteps": 240.76470588235293,
"averageOutputTokens": 59167.56470588235,
"averageTotalCostUsd": 7.338732845647058,
"averageOutputCostUsd": 0.8875134705882353
}Evidence observed Jul 22, 2026 | 12.9 |
| 22 | Kimi K2.6 (Reasoning) source senior_swe_bench · Kimi K2.6 · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider moonshotai/kimi-k2.6 · record kimi-k2-6-mini-swe · run 8b3a3d926c6f93221f2f3d40f0007ec45548bfc629a850c32391fa9e9fc7437d View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:kimi-k2.6",
"basicPassAt1": 24.705882352941178,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 9.411764705882353,
"agentDescription": "Moonshot Kimi K2.6 via mini-swe-agent harness (Portkey→OpenRouter)",
"privateTaskCount": 50,
"upstreamProvider": "kimi",
"upstreamRunCount": 50,
"averageAgentSteps": 221.73863636363637,
"averageOutputTokens": 513130.6477272727,
"averageTotalCostUsd": 2.6998195050681817,
"averageOutputCostUsd": 1.74977550875
}Evidence observed Jul 1, 2026 | 9.4 |
| 23 | Claude Sonnet 4.6 (Non-reasoning, High) source senior_swe_bench · Claude Sonnet 4.6 (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider anthropic/claude-sonnet-4-6 · record sonnet-mini-swe · run af85cd5c6c05730a67b213de74628afcdcfe842a6c59e3f347f9c7240081be3a View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:claude-sonnet-4-6",
"basicPassAt1": 30.23255813953488,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 9.30232558139535,
"agentDescription": "Claude Sonnet 4.6 via mini-swe-agent harness (Portkey→Bedrock)",
"privateTaskCount": 50,
"upstreamProvider": "anthropic",
"upstreamRunCount": 80,
"averageAgentSteps": 175.2528735632184,
"averageOutputTokens": 61075.252873563215,
"averageTotalCostUsd": 5.490449816551724,
"averageOutputCostUsd": 0.9161287931034483
}Evidence observed Jun 30, 2026 | 9.3 |
| 24 | GPT-5.6 Luna (Low) source senior_swe_bench · GPT-5.6 Luna (low) · low · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider openai/gpt-5.6-luna · record gpt-5-6-luna-low-mini-swe · run 24265362e37e7833d388a5ca1d19b0ac31db458e5214082c8cf44f0f56e46e61 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gpt-5.6-luna",
"basicPassAt1": 20.27027027027027,
"basicPassAt3": 25.675675675675674,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 8.108108108108109,
"tastefulPassAt3": 13.513513513513514,
"agentDescription": "OpenAI gpt-5.6-luna via mini-swe-agent at reasoning_effort=low (leaderboard-v2 sweep)",
"privateTaskCount": 50,
"upstreamProvider": "openai",
"upstreamRunCount": 38,
"averageAgentSteps": 13.64864864864865,
"averageOutputTokens": 2590.3783783783783,
"averageTotalCostUsd": 0.029366041081081082,
"averageOutputCostUsd": 0.015542270270270272,
"basicAllThreePassRate": 13.513513513513514,
"tastefulAllThreePassRate": 5.405405405405405
}Evidence observed Jul 12, 2026 | 8.1 |
| 25 | Gemini 3.1 Pro Preview source senior_swe_bench · Gemini 3.1 Pro (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider google/gemini-3.1-pro-preview · record gemini-3-1-pro-mini-swe · run 3018be1dee8e4b352d3aa9260d805ee298040ab383dec290a16294f9afd8472d View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gemini-3.1-pro-preview",
"basicPassAt1": 26.136363636363637,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 6.8181818181818175,
"agentDescription": "Google Gemini 3.1 Pro via mini-swe-agent harness (Portkey→Google)",
"privateTaskCount": 50,
"upstreamProvider": "google",
"upstreamRunCount": 50,
"averageAgentSteps": 89.52272727272727,
"averageOutputTokens": 19415.81818181818,
"averageTotalCostUsd": 0.8407021872727273,
"averageOutputCostUsd": 0.23298981818181816
}Evidence observed Jun 30, 2026 | 6.8 |
| 26 | Gemini 3.5 Flash (High) source senior_swe_bench · Gemini 3.5 Flash (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider google/gemini-3.5-flash · record gemini-3-5-flash-mini-swe · run 81a3966a96064929ee903c851f910db97c925b6374c977b4e44de27195ae9c35 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gemini-3.5-flash",
"basicPassAt1": 22.105263157894736,
"basicPassAt3": 38.94736842105263,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 6.315789473684211,
"tastefulPassAt3": 17.894736842105264,
"agentDescription": "Google Gemini 3.5 Flash via mini-swe-agent harness (Portkey→Google)",
"privateTaskCount": 50,
"upstreamProvider": "google",
"upstreamRunCount": 80,
"averageAgentSteps": 230.83870967741936,
"averageOutputTokens": 70372.86021505376,
"averageTotalCostUsd": 4.368946809032258,
"averageOutputCostUsd": 0.6333557419354838,
"basicAllThreePassRate": 13.684210526315791,
"tastefulAllThreePassRate": 2.1052631578947367
}Evidence observed Jul 29, 2026 | 6.3 |
| 27 | Gemini 3.5 Flash (Medium) source senior_swe_bench · Gemini 3.5 Flash (medium) · medium · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider google/gemini-3.5-flash · record gemini-3-5-flash-mini-swe · run 47baaf6e0c18a44b9752a875a974ff5649b1390497d94a2a516eb5cae8047068 View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gemini-3.5-flash",
"basicPassAt1": 20.454545454545457,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 3.4090909090909087,
"agentDescription": "Google Gemini 3.5 Flash via mini-swe-agent harness (Portkey→Google)",
"privateTaskCount": 50,
"upstreamProvider": "google",
"upstreamRunCount": 80,
"averageAgentSteps": 255.1590909090909,
"averageOutputTokens": 84979.35227272728,
"averageTotalCostUsd": 6.14804299909091,
"averageOutputCostUsd": 0.7648141704545455
}Evidence observed Jun 30, 2026 | 3.4 |
| 28 | Gemini 3.1 Pro Preview source senior_swe_bench · Gemini 3.1 Pro (high) · high · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider google/gemini-3.1-pro-preview · record gemini-3-1-pro-mini-swe · run e7a084b8bf4fbf7e4723f412d7fc0b621bc5db20fd0e61b5014f602fa22d436e View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:gemini-3.1-pro-preview",
"basicPassAt1": 9.473684210526317,
"basicPassAt3": 24.210526315789473,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 2.1052631578947367,
"tastefulPassAt3": 8.421052631578947,
"agentDescription": "Google Gemini 3.1 Pro via mini-swe-agent harness (Portkey→Google)",
"privateTaskCount": 50,
"upstreamProvider": "google",
"upstreamRunCount": 50,
"averageAgentSteps": 107.73684210526316,
"averageOutputTokens": 17277.894736842107,
"averageTotalCostUsd": 1.3306002509473684,
"averageOutputCostUsd": 0.2073347368421053,
"basicAllThreePassRate": 5.263157894736842,
"tastefulAllThreePassRate": 0
}Evidence observed Jul 29, 2026 | 2.1 |
| 29 | Inkling (Xhigh) source senior_swe_bench · Inkling (max) · max · mini-swe-agent · benchmark 2026.06 · split official-100-task · 1 attempts · pass@1 · harbor:snorkel-ai/senior-swe-bench-v2026.06 · provider thinkingmachines/Inkling · record thinkingmachines-inkling-mini-swe · run b927c3ef29f49a1dc839e038f8b3cd3b4a201370e95bb0a8d032a9a0e129a40e View run sourceSource metadata{
"taskCount": 100,
"agentGroup": "mini-swe-agent:Inkling",
"basicPassAt1": 3.1578947368421053,
"basicPassAt3": 5.263157894736842,
"datasetPackage": "snorkel-ai/senior-swe-bench-v2026.06",
"publicTaskCount": 50,
"tastefulPassAt1": 0,
"tastefulPassAt3": 0,
"agentDescription": "Thinking Machines Inkling via mini-swe-agent harness (Portkey→Together.ai); reasoning max",
"privateTaskCount": 50,
"upstreamProvider": null,
"upstreamRunCount": 50,
"averageAgentSteps": 51.2,
"averageOutputTokens": 3398.1473684210528,
"averageTotalCostUsd": 0,
"averageOutputCostUsd": 0,
"basicAllThreePassRate": 1.0526315789473684,
"tastefulAllThreePassRate": 0
}Evidence observed Jul 23, 2026 | 0.0 |
Descriptive observations, not a causal comparison or routing recommendation. Preliminary and unknown-effort rows are excluded. Terminal-Bench totals may include disqualified trials; that accounting is unverified.
| Model / effort | Score [%] | Cost [USD per task] | Cost basis | Time [s per attempt] | Observed upstream | Source / run |
|---|---|---|---|---|---|---|
| Gemini 3.5 Flash (medium) · medium | 3.4 | $6.15 | Source-reported average (accounting unverified; not verified measured cost) | Not published / unit unverified | 2026-06-30 | gemini-3-5-flash-mini-swe 47baaf6e0c18a44b9752a875a974ff5649b1390497d94a2a516eb5cae8047068 |
| Gemini 3.5 Flash (high) · high | 6.3 | $4.37 | Source-reported average (accounting unverified; not verified measured cost) | Not published / unit unverified | 2026-07-29 | gemini-3-5-flash-mini-swe 81a3966a96064929ee903c851f910db97c925b6374c977b4e44de27195ae9c35 |