Chemical knowledge and reasoning benchmark spanning nine topics: analytical, general, inorganic, organic, physical, and technical chemistry plus materials science, toxicology, and chemical preference. Published in Nature Chemistry (2025) by the LamaLab at the University of Jena.
This definition has not completed editorial and provenance review. Verify its metric, version, source, and evaluation configuration upstream before using it for a model-selection decision.
Domain-expert curated questions with topic-level breakdowns; published in a top-tier journal; covers applied areas (toxicology, materials) that general science benchmarks miss.
Static dataset with moderate contamination risk (items from textbooks and exam banks); overall score conflates very different cognitive demands across topics; React-agent variants inflate leaderboard size without necessarily reflecting base-model skill.
Higher is better. Overall accuracy (0–100%) averaged across the nine topic scores.
Static dataset with moderate contamination risk (items from textbooks and exam banks).
| # | Model | Score |
|---|---|---|
| 1 | 65.8 | |
| 2 | 65.8 | |
| 3 | 64.0 | |
| 4 | 63.0 | |
| 5 | 61.0 | |
| 6 | 58.0 | |
| 7 | 58.0 | |
| 8 | 57.0 | |
| 9 | 57.0 | |
| 10 | 53.0 | |
| 11 | 52.0 | |
| 12 | 52.0 | |
| 13 | 51.0 | |
| 14 | 48.0 | |
| 15 | 47.0 | |
| 16 | 47.0 | |
| 17 | 47.0 | |
| 18 | 47.0 | |
| 19 | 46.0 | |
| 20 | 46.0 | |
| 21 | 45.0 | |
| 22 | 45.0 | |
| 23 | 45.0 | |
| 24 | 42.0 | |
| 25 | 27.0 | |
| 26 | 26.0 | |
| 27 | 0.6 |