BudHist Benchmark - Testing the Knowledge of Buddhist History Encoded in Open-weights Language Models

This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.
We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, Ollama is used as middle-ware for the local experiments, OpenRouter for the cloud-based tests.
The aim of the project is to:

As of , among the tested local open-weight models qwen3.5:35b and qwen3.6:35b 'know' most about Buddhist history. Among the tested cloud-based open-weights models the gemini and glm series are slightly ahead, with the deepseek and kimi series close behind. The best open-weight, cloud-based models have basically saturated the benchmark (90%+). The closed, non-open-weights models we included as reference perform in the same ball park.

Marcus Bingenheimer

Oct 2025 - now


Model Test 2025-10
(150 questions)
Test 2026-04
(210 questions)
Test 2026-08
(250 questions)
Test 2026-11
(260 questions)

Local
deepseek-r1:14b 128/210 (61%)
gemma2:9b 81/150 (54%)
gemma3:4b 60/150 (40%)
gemma3:12b 93/150 (62%) 134/210 (64%) 171/250 (68%)
gemma3:27b 96/150 (64%) 139/210 (66%)
gemma4:e4b 121/210 (58%)
gemma4:31b 159/210 (76%) 198/250 (79%) 205/260 (79%)
glm-4.7-flash:q4_K_M (latest) 158/210 (75%) 176/250 (70%)
glm-4.7-flash:q8_0 205/260 (79%)
laguna-xs-2.1:q4_K_M 146/250 (58%)
llama3:8b 84/150 (56%)
llama3.1:8b 85/150 (57%)
llama4:16x17b X[n] needs 60GB local memory
mistral:7b 72/150 (48%)
mistral-small3.1:24b 203/260 (78%)
mistral-nemo:12b 124/210 (59%) 154/250 (62%) 154/260 (59%)
nehrdbsd/mitra-chat (Gemma 2 based) 162/250 (65%)
nehrdbsd/mitra-chat-qwen (qwen3.5 based) 189/250 (76%)[n] Mitra is optimized for translation not general knowledge. See online application here: https://dharmamitra.org/translate 206/260 (79%)
mixtral:8x7b 98/150 (65%) 151/210 (72%)
nemotron-3.5-lightning:30b 184/260 (71%)
olmo-3:7b X[n] Olmo does not take “think=False”. Workaround slows the response time drastically.
olmo-3.1:32b X[n] Olmo does not take “think=False”. Workaround slows the response time drastically.
ornith-1.5:35b 213/260 (82%)
phi3:3.8b 75/150 (50%)
phi4:14b (latest) 81/150 (54%) 132/210 (63%) 168/250 (67%) 175/260 (67%)
qwen2.5:7b 86/150 (57%)
qwen2.5:14b 114/150 (76%) 158/210 (75%) 196/250 (78%)
qwen2.5:32b 120/150 (80%) 165/210 (79%) 200/250 (80%)
qwen3:8b 102/150 (68%)
qwen3:14b 114/150 (76%) 158/210 (75%) 194/250 (78%)
qwen3:32b 110/150 (73%)
qwen3.5:9b (latest) 153/210 (73%) 185/250 (74%)
qwen3.5:27b 180/210 (86%) 205/250 (82%)
qwen3.5:35b 221/260 (85%)
qwen3.6:35b (latest) 210/250 (84%) 218/260 (84%)
qwen3.8:27b 198/250 (79%) 207/260 (80%)
Cloud (Ollama Cloud) (OpenRouter) (OpenRouter)
deepseek-v3.2 191/210 (91%)
deepseek-v4-flash 230/250 (92%)
deepseek-v4.1-flash-1017 244/260 (94%) (12.0m)
deepseek-v4-pro 232/250 (93%) 240/260 (92%) (33.7m)
devstral-2:123b 171/210 (81%)
gemini-3.5-flash-lite 229/250 (92%)
gemini-3.6-flash 233/250 (93%) 242/260 (93%) (9.4m)
gemini-3.8-flash 246/260 (95%) (41.5m)
gemini-4-argon 245/260 (94%) (23.7m)
glm-4.7 188/210 (90%)
glm-5.1 192/210 (91%)
glm-5.2 238/250 (95%) 245/260 (94%) (18.2m)
glm-5.3-flash 239/260 (92%) (16.1m)
glm-5.3 238/260 (92%) (12.8m)
gpt-oss-120b 77/210 (37%) 199/250 (80%)
hy3 229/250 (92%)
hy4 196/260 (75%) (62 Errors, 143.6m)
inkling 231/250 (92%) 242/260 (93%) (125.8m)
inkling-small 221/250 (88%) 233/260 (90%) (25.5m)
kimi-k2-thinking 197/210 (94%) 231/250 (92%)
kimi-k2.5 187/210 (89%)
kimi-k2.6 230/250 (92%) 237/260 (91%) (84.4m)
kimi-k3.1-pro 244/260 (94%) (31.1m)
laguna-s-2.1 X[n] No ZDR endpoint
minimax-m3 234/250 (94%) 236/260 (91%) (15.8m)
minimax-m3.1-flash 215/260 (83%) (15.8m)
mistral-large-2512 185/210 (88%)
mistral-small-2603 202/250 (81%)
mistral-medium-3-5 X[n] No ZDR endpoint.
mistral-large-4-0 X[n] No ZDR endpoint.
qwen3.5:397b 187/210 (89%)
qwen3.5-9b 200/250 (80%)
qwen3.6-35b-a3b 217/250 (87%)
qwen3.8-max-0902 X[n] No ZDR endpoint.
Closed Models (OpenRouter) (OpenRouter)
claude-sonnet-4-5 230/250 (92%)
claude-sonnet-4-6 233/250 (93%) 243/260 (93%) (9.1m)
claude-sonnet-5-5 229/260 (88%) (25 Errors, 30.4m)
gpt-4.1 216/250 (86%)
gpt-5.6-luna 222/250 (89%)
gpt-6-luna 234/260 (90%) (16 Errors, 30.4m)

Observations

Comparing 2026-08 with 2026-04 results

Comparing 2026-08 with 2026-04 results

Comparing 2026-04 with 2025-10 results


[home]