MultiChallenge evaluates multi-turn conversation reliability across instruction retention, inference memory, versioned editing, and self-coherence.
Tests real multi-turn conversation failures; highly unsaturated; directly relevant to chatbot quality.
Synthetic scenarios do not capture the full distribution of production conversations. Scores also depend on the conversation protocol and turn budget, which must be preserved for comparison.
Higher is better. Percentage of multi-turn challenges completed correctly.
Synthetic scenarios do not capture the full distribution of production conversations.
| # | Model | Score |
|---|---|---|
| 1 | 85.9 | |
| 2 | 82.0 | |
| 3 | 82.0 | |
| 4 | 80.6 | |
| 5 | 80.4 | |
| 6 | 78.7 | |
| 7 | 75.5 | |
| 8 | 75.3 | |
| 9 | 74.2 | |
| 10 | 71.4 | |
| 11 | 69.2 | |
| 12 | 67.7 | |
| 13 | 65.7 | |
| 14 | 63.4 | |
| 15 | 63.2 |