This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.
We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, Ollama is used as middle-ware for the local experiments, OpenRouter for the cloud-based tests.
The aim of the project is to:
As of , among the tested local open-weight models qwen3.6:35b 'knows' most about Buddhist history. Among the tested cloud-based open-weights models glm-5.2 and minimax-m3 are best, but with deepseek-v4-pro and gemini-3.6-flash tied close behind. With that (~94%) the best open-weight, cloud-based models have basically saturated the benchmark. The closed, non-open-weights models we included as reference perform in the same ball park.
April 2026 - now
| Model | Test 2025-10 (150 questions) | Test 2026-04 (210 questions) | Test 2026-08 (250 questions) |
|---|---|---|---|
| Local | |||
| deepseek-r1:14b | 128/210 (61%) | ||
| gemma2:9b | 81/150 (54%) | ||
| gemma3:4b | 60/150 (40%) | ||
| gemma3:12b | 93/150 (62%) | 134/210 (64%) | 171/250 (68%) |
| gemma3:27b | 96/150 (64%) | 139/210 (66%) | |
| gemma4:e4b | 121/210 (58%) | ||
| gemma4:31b | 159/210 (76%) | 198/250 (79%) | |
| glm-4.7-flash:q4_K_M (latest) | 158/210 (75%) | 176/250 (70%) | |
| laguna-xs-2.1:q4_K_M | 146/250 (58%) | ||
| llama3:8b | 84/150 (56%) | ||
| llama3.1:8b | 85/150 (57%) | ||
| llama4:16x17b | X[n] needs 60GB local memory | ||
| mistral:7b | 72/150 (48%) | ||
| mistral-nemo:12b | 124/210 (59%) | 154/250 (62%) | |
| nehrdbsd/mitra-chat (Gemma 2 based) | 162/250 (65%) | ||
| nehrdbsd/mitra-chat-qwen (qwen3.5 based) | 189/250 (76%)[n] Mitra is optimized for translation not general knowledge. See online application here: https://dharmamitra.org/translate | ||
| mixtral:8x7b | 98/150 (65%) | 151/210 (72%) | |
| olmo-3:7b | X[n] Olmo does not take “think=False”. Workaround slows the response time drastically. | ||
| olmo-3.1:32b | X[n] Olmo does not take “think=False”. Workaround slows the response time drastically. | ||
| phi3:3.8b | 75/150 (50%) | ||
| phi4:14b (latest) | 81/150 (54%) | 132/210 (63%) | 168/250 (67%) |
| qwen2.5:7b | 86/150 (57%) | ||
| qwen2.5:14b | 114/150 (76%) | 158/210 (75%) | 196/250 (78%) |
| qwen2.5:32b | 120/150 (80%) | 165/210 (79%) | 200/250 (80%) |
| qwen3:8b | 102/150 (68%) | ||
| qwen3:14b | 114/150 (76%) | 158/210 (75%) | 194/250 (78%) |
| qwen3:32b | 110/150 (73%) | ||
| qwen3.5:9b (latest) | 153/210 (73%) | 185/250 (74%) | |
| qwen3.5:27b | 180/210 (86%) | 205/250 (82%) | |
| qwen3.6:35b | 210/250 (84%) | ||
| Cloud | (Ollama Cloud) | (OpenRouter) | |
| deepseek-v3.2 | 191/210 (91%) | ||
| deepseek-v4-flash | 230/250 (92%) | ||
| deepseek-v4-pro | 232/250 (93%) | ||
| devstral-2:123b | 171/210 (81%) | ||
| gemini-3.5-flash-lite | 229/250 (92%) | ||
| gemini-3.6-flash | 233/250 (93%) | ||
| glm-4.7 | 188/210 (90%) | ||
| glm-5.1 | 192/210 (91%) | ||
| glm-5.2 | 238/250 (95%) | ||
| gpt-oss-120b | 77/210 (37%) | 199/250 (80%) | |
| hy3 | 229/250 (92%) | ||
| inkling-small | 221/250 (88%) | ||
| inkling | 231/250 (92%) | ||
| kimi-k2-thinking | 197/210 (94%) | 231/250 (92%) | |
| kimi-k2.5 | 187/210 (89%) | ||
| kimi-k2.6 | 230/250 (92%) | ||
| laguna-s-2.1 | X[n] No ZDR endpoint | ||
| minimax-m3 | 234/250 (94%) | ||
| mistral-large-2512 | 185/210 (88%) | ||
| mistral-small-2603 | 202/250 (81%) | ||
| mistral-medium-3-5 | X[n] No ZDR endpoint. | ||
| qwen3.5:397b | 187/210 (89%) | ||
| qwen3.5-9b | 200/250 (80%) | ||
| qwen3.6-35b-a3b | 217/250 (87%) | ||
| Closed Models | (OpenRouter) | ||
| claude-sonnet-4-5 | 230/250 (92%) | ||
| claude-sonnet-4-6 | 233/250 (93%) | ||
| gpt-4.1 | 216/250 (86%) | ||
| gpt-5.6-luna | 222/250 (89%) |