Tests LLMs on complex terminal/CLI tasks — system administration, scripting, file manipulation, and command-line problem solving. Version 2.0 includes harder, multi-step tasks.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Tests practical CLI and sysadmin skills that coding benchmarks miss. Real-world applicable.
Wide score range (3–69%) reflects high difficulty variance. Version 2.0 is significantly harder than v1.
Higher is better. Task completion rate (0–100%).
Wide score range (3–69%) reflects high difficulty variance.
| # | Model | Score |
|---|---|---|
| 1 | 99.9 | |
| 2 | 99.8 | |
| 3 | 99.5 | |
| 4 | 91.6 | |
| 5 | 91.2 | |
| 6 | 91.0 | |
| 7 | 90.8 | |
| 8 | 90.7 | |
| 9 | 88.2 | |
| 10 | 87.4 | |
| 11 | 87.3 | |
| 12 | 85.8 | |
| 13 | 85.2 | |
| 14 | 84.9 | |
| 15 | 83.4 | |
| 16 | 83.2 | |
| 17 | 82.3 | |
| 18 | 82.0 | |
| 19 | 81.9 | |
| 20 | 81.0 | |
| 21 | 79.7 | |
| 22 | 79.6 | |
| 23 | 79.6 | |
| 24 | 79.5 | |
| 25 | 79.3 | |
| 26 | 78.3 | |
| 27 | 77.4 | |
| 28 | 76.9 | |
| 29 | 76.9 | |
| 30 | 76.8 | |
| 31 | 76.4 | |
| 32 | 75.7 | |
| 33 | 75.2 | |
| 34 | 75.1 | |
| 35 | 74.0 | |
| 36 | 73.8 | |
| 37 | 73.7 | |
| 38 | 71.9 | |
| 39 | 71.6 | |
| 40 | 71.5 | |
| 41 | 71.4 | |
| 42 | 71.3 | |
| 43 | 71.2 | |
| 44 | 69.9 | |
| 45 | 69.4 | |
| 46 | 69.0 | |
| 47 | 68.7 | |
| 48 | 68.7 | |
| 49 | 68.6 | |
| 50 | 68.5 |