Large-scale Winograd Schema Challenge — tests commonsense reasoning through pronoun disambiguation in sentences that require world knowledge to resolve.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Tests genuine commonsense understanding beyond surface heuristics. Adversarially filtered to remove bias.
Score is 0–1 ratio. Approaching saturation (top ~0.89). In training data for most models.
Higher is better. Accuracy ratio (0–1). Random baseline: 0.5 (binary choice).
Score is 0–1 ratio.
| # | Model | Score |
|---|---|---|
| 1 | 1.6 | |
| 2 | 1.6 | |
| 3 | 1.5 | |
| 4 | 1.5 | |
| 5 | 1.4 | |
| 6 | 1.4 | |
| 7 | 1.4 | |
| 8 | 1.4 | |
| 9 | 1.3 | |
| 10 | 1.3 | |
| 11 | 1.3 | |
| 12 | 1.2 | |
| 13 | 1.2 | |
| 14 | 1.2 | |
| 15 | 1.2 | |
| 16 | 1.2 | |
| 17 | 1.2 | |
| 18 | 1.2 | |
| 19 | 1.2 | |
| 20 | 1.2 | |
| 21 | 1.2 | |
| 22 | 1.2 | |
| 23 | 1.1 | |
| 24 | 1.1 | |
| 25 | 1.1 | |
| 26 | 1.1 | |
| 27 | 1.1 | |
| 28 | 1.1 | |
| 29 | 1.1 | |
| 30 | 1.1 | |
| 31 | 1.1 | |
| 32 | 1.1 | |
| 33 | 1.1 | |
| 34 | 1.1 | |
| 35 | 1.1 | |
| 36 | 1.1 | |
| 37 | 1.1 | |
| 38 | 1.1 | |
| 39 | 1.1 | |
| 40 | 1.1 | |
| 41 | 1.0 | |
| 42 | 1.0 | |
| 43 | 1.0 | |
| 44 | 1.0 | |
| 45 | 1.0 | |
| 46 | 1.0 | |
| 47 | 1.0 | |
| 48 | 1.0 | |
| 49 | 1.0 | |
| 50 | 1.0 |