This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.
We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, Ollama is used as middle-ware for the local experiments, OpenRouter for the cloud-based tests.
The aim of the project is to:
As of , among the tested local open-weight models qwen3.5:35b and qwen3.6:35b 'know' most about Buddhist history. Among the tested cloud-based open-weights models the gemini and glm series are slightly ahead, with the deepseek and kimi series close behind. The best open-weight, cloud-based models have basically saturated the benchmark (90%+). The closed, non-open-weights models we included as reference perform in the same ball park.
Oct 2025 - now
| Model | Test 2025-10 (150 questions) |
Test 2026-04 (210 questions) |
Test 2026-08 (250 questions) |
Test 2026-11 (260 questions) |
|---|---|---|---|---|
| Local | ||||
| deepseek-r1:14b | 128/210 (61%) | |||
| gemma2:9b | 81/150 (54%) | |||
| gemma3:4b | 60/150 (40%) | |||
| gemma3:12b | 93/150 (62%) | 134/210 (64%) | 171/250 (68%) | |
| gemma3:27b | 96/150 (64%) | 139/210 (66%) | ||
| gemma4:e4b | 121/210 (58%) | |||
| gemma4:31b | 159/210 (76%) | 198/250 (79%) | 205/260 (79%) | |
| glm-4.7-flash:q4_K_M (latest) | 158/210 (75%) | 176/250 (70%) | ||
| glm-4.7-flash:q8_0 | 205/260 (79%) | |||
| laguna-xs-2.1:q4_K_M | 146/250 (58%) | |||
| llama3:8b | 84/150 (56%) | |||
| llama3.1:8b | 85/150 (57%) | |||
| llama4:16x17b | X[n] needs 60GB local memory | |||
| mistral:7b | 72/150 (48%) | |||
| mistral-small3.1:24b | 203/260 (78%) | |||
| mistral-nemo:12b | 124/210 (59%) | 154/250 (62%) | 154/260 (59%) | |
| nehrdbsd/mitra-chat (Gemma 2 based) | 162/250 (65%) | |||
| nehrdbsd/mitra-chat-qwen (qwen3.5 based) | 189/250 (76%)[n] Mitra is optimized for translation not general knowledge. See online application here: https://dharmamitra.org/translate | 206/260 (79%) | ||
| mixtral:8x7b | 98/150 (65%) | 151/210 (72%) | ||
| nemotron-3.5-lightning:30b | 184/260 (71%) | |||
| olmo-3:7b | X[n] Olmo does not take “think=False”. Workaround slows the response time drastically. | |||
| olmo-3.1:32b | X[n] Olmo does not take “think=False”. Workaround slows the response time drastically. | |||
| ornith-1.5:35b | 213/260 (82%) | |||
| phi3:3.8b | 75/150 (50%) | |||
| phi4:14b (latest) | 81/150 (54%) | 132/210 (63%) | 168/250 (67%) | 175/260 (67%) |
| qwen2.5:7b | 86/150 (57%) | |||
| qwen2.5:14b | 114/150 (76%) | 158/210 (75%) | 196/250 (78%) | |
| qwen2.5:32b | 120/150 (80%) | 165/210 (79%) | 200/250 (80%) | |
| qwen3:8b | 102/150 (68%) | |||
| qwen3:14b | 114/150 (76%) | 158/210 (75%) | 194/250 (78%) | |
| qwen3:32b | 110/150 (73%) | |||
| qwen3.5:9b (latest) | 153/210 (73%) | 185/250 (74%) | ||
| qwen3.5:27b | 180/210 (86%) | 205/250 (82%) | ||
| qwen3.5:35b | 221/260 (85%) | |||
| qwen3.6:35b (latest) | 210/250 (84%) | 218/260 (84%) | ||
| qwen3.8:27b | 198/250 (79%) | 207/260 (80%) | ||
| Cloud | (Ollama Cloud) | (OpenRouter) | (OpenRouter) | |
| deepseek-v3.2 | 191/210 (91%) | |||
| deepseek-v4-flash | 230/250 (92%) | |||
| deepseek-v4.1-flash-1017 | 244/260 (94%) (12.0m) | |||
| deepseek-v4-pro | 232/250 (93%) | 240/260 (92%) (33.7m) | ||
| devstral-2:123b | 171/210 (81%) | |||
| gemini-3.5-flash-lite | 229/250 (92%) | |||
| gemini-3.6-flash | 233/250 (93%) | 242/260 (93%) (9.4m) | ||
| gemini-3.8-flash | 246/260 (95%) (41.5m) | |||
| gemini-4-argon | 245/260 (94%) (23.7m) | |||
| glm-4.7 | 188/210 (90%) | |||
| glm-5.1 | 192/210 (91%) | |||
| glm-5.2 | 238/250 (95%) | 245/260 (94%) (18.2m) | ||
| glm-5.3-flash | 239/260 (92%) (16.1m) | |||
| glm-5.3 | 238/260 (92%) (12.8m) | |||
| gpt-oss-120b | 77/210 (37%) | 199/250 (80%) | ||
| hy3 | 229/250 (92%) | |||
| hy4 | 196/260 (75%) (62 Errors, 143.6m) | |||
| inkling | 231/250 (92%) | 242/260 (93%) (125.8m) | ||
| inkling-small | 221/250 (88%) | 233/260 (90%) (25.5m) | ||
| kimi-k2-thinking | 197/210 (94%) | 231/250 (92%) | ||
| kimi-k2.5 | 187/210 (89%) | |||
| kimi-k2.6 | 230/250 (92%) | 237/260 (91%) (84.4m) | ||
| kimi-k3.1-pro | 244/260 (94%) (31.1m) | |||
| laguna-s-2.1 | X[n] No ZDR endpoint | |||
| minimax-m3 | 234/250 (94%) | 236/260 (91%) (15.8m) | ||
| minimax-m3.1-flash | 215/260 (83%) (15.8m) | |||
| mistral-large-2512 | 185/210 (88%) | |||
| mistral-small-2603 | 202/250 (81%) | |||
| mistral-medium-3-5 | X[n] No ZDR endpoint. | |||
| mistral-large-4-0 | X[n] No ZDR endpoint. | |||
| qwen3.5:397b | 187/210 (89%) | |||
| qwen3.5-9b | 200/250 (80%) | |||
| qwen3.6-35b-a3b | 217/250 (87%) | |||
| qwen3.8-max-0902 | X[n] No ZDR endpoint. | |||
| Closed Models | (OpenRouter) | (OpenRouter) | ||
| claude-sonnet-4-5 | 230/250 (92%) | |||
| claude-sonnet-4-6 | 233/250 (93%) | 243/260 (93%) (9.1m) | ||
| claude-sonnet-5-5 | 229/260 (88%) (25 Errors, 30.4m) | |||
| gpt-4.1 | 216/250 (86%) | |||
| gpt-5.6-luna | 222/250 (89%) | |||
| gpt-6-luna | 234/260 (90%) (16 Errors, 30.4m) |