Conversational agent benchmark — simulated airline customer service tasks measuring tool-calling accuracy, policy adherence, and multi-turn dialogue reliability.
Tests policy compliance, tool-calling correctness, and multi-turn state tracking in a realistic service workflow; domain-specific constraints expose failure modes hidden by generic chat benchmarks.
Limited to airline domain. Pass^1 metric may not capture partial task completion.
Higher is better. Pass^1 rate (percentage of tasks completed correctly on first attempt).
Max score: 100
Limited to airline domain.
| # | Model | Score |
|---|---|---|
| 1 | 42.0 | |
| 2 | 42.0 | |
| 3 | 36.0 | |
| 4 | 22.5 |