Adyen's Data Analysis Benchmark (Step). Multi-step agentic workflow evaluation: models must iterate through SQL and Python to answer business questions against realistic payment-industry datasets. Hard-level accuracy is the canonical score; Easy is provided for context.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Real-world data analysis tasks with grounded success criteria; agent frameworks are open-sourced in the Space; Easy/Hard split exposes ceiling behavior of top models.
Custom-agent submissions dominate the board — a model's reported score reflects the submission framework's prompting, tool use, and retries as much as the base-model capability; benchmark is small (~49 submissions) so single-submission variance is high.
Higher is better. Hard-level accuracy (%) — fraction of multi-step business questions answered correctly.
Custom-agent submissions dominate the board — a model's reported score reflects the submission framework's prompting, tool use, and retries as much as the base-model capability.
| # | Model | Score |
|---|---|---|
| 1 | 100.0 | |
| 2 | 98.9 | |
| 3 | 94.4 | |
| 4 | 94.2 | |
| 5 | 90.0 | |
| 6 | 88.6 | |
| 7 | 57.7 | |
| 8 | 40.5 | |
| 9 | 28.0 | |
| 10 | 19.8 | |
| 11 | 19.8 | |
| 12 | 14.6 | |
| 13 | 14.6 | |
| 14 | 13.8 | |
| 15 | 13.8 | |
| 16 | 13.8 | |
| 17 | 13.8 | |
| 18 | 12.7 | |
| 19 | 12.4 | |
| 20 | 12.4 | |
| 21 | 11.1 | |
| 22 | 11.1 | |
| 23 | 9.8 | |
| 24 | 9.8 | |
| 25 | 9.3 | |
| 26 | 9.0 | |
| 27 | 9.0 | |
| 28 | 8.7 | |
| 29 | 6.1 | |
| 30 | 5.6 | |
| 31 | 5.0 | |
| 32 | 3.7 | |
| 33 | 3.7 | |
| 34 | 3.4 | |
| 35 | 1.9 | |
| 36 | 0.0 | |
| 37 | 0.0 |