Graduate-Level Google-Proof Q&A (Diamond subset) as reported in the LLM Benchmark Matrix. Expert-crafted questions from biology, physics, and chemistry that are deliberately hard to Google.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Tests genuine expert-level scientific reasoning. Questions are validated to be un-Googleable.
Matrix snapshot — we also have AA's own GPQA column. Small question set (198 questions in Diamond subset).
Higher is better. Accuracy percentage (0–100%). Human expert baseline ~65%.
Matrix snapshot — we also have AA's own GPQA column.
| # | Model | Score |
|---|---|---|
| 1 | 94.6 | |
| 2 | 94.5 | |
| 3 | 94.4 | |
| 4 | 94.3 | |
| 5 | 94.3 | |
| 6 | 93.3 | |
| 7 | 93.3 | |
| 8 | 93.2 | |
| 9 | 93.2 | |
| 10 | 93.2 | |
| 11 | 93.2 | |
| 12 | 92.9 | |
| 13 | 92.9 | |
| 14 | 92.8 | |
| 15 | 92.7 | |
| 16 | 92.5 | |
| 17 | 92.4 | |
| 18 | 92.1 | |
| 19 | 92.0 | |
| 20 | 91.9 | |
| 21 | 91.9 | |
| 22 | 91.9 | |
| 23 | 91.8 | |
| 24 | 91.5 | |
| 25 | 91.3 | |
| 26 | 91.3 | |
| 27 | 91.3 | |
| 28 | 91.2 | |
| 29 | 91.1 | |
| 30 | 91.1 | |
| 31 | 91.0 | |
| 32 | 91.0 | |
| 33 | 91.0 | |
| 34 | 91.0 | |
| 35 | 90.8 | |
| 36 | 90.7 | |
| 37 | 90.7 | |
| 38 | 90.5 | |
| 39 | 90.5 | |
| 40 | 90.4 | |
| 41 | 90.4 | |
| 42 | 90.0 | |
| 43 | 90.0 | |
| 44 | 89.8 | |
| 45 | 89.8 | |
| 46 | 89.7 | |
| 47 | 89.7 | |
| 48 | 89.4 | |
| 49 | 89.4 | |
| 50 | 89.2 |