Evaluates LLMs on practical, challenging coding tasks that require composing multiple function calls and libraries — closer to real-world software development than HumanEval.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Tests realistic multi-step coding with library usage. Higher difficulty ceiling than HumanEval.
Relatively new benchmark with fewer evaluated models. Scores cluster in a narrow range (34–50%).
Higher is better. Pass rate percentage (0–100%).
Relatively new benchmark with fewer evaluated models.
| # | Model | Score |
|---|---|---|
| 1 | 50.0 | |
| 2 | 49.7 | |
| 3 | 48.9 | |
| 4 | 45.9 | |
| 5 | 45.9 | |
| 6 | 45.9 | |
| 7 | 45.5 | |
| 8 | 44.6 | |
| 9 | 33.8 |