AdvaitaBench

tasks 84
families 8
judges 2
models 10

Most models can sound profound. Few can hold a precise position when you argue back.

A closed-book eval of classical Advaita Vedānta: school boundaries, levels of reality, text-grounded reading, and multi-turn pressure. Built to surface confident vagueness and doctrine that flips — the reliability failures that trivia benchmarks miss.

View leaderboard → Read the write-up

Leaderboard

The headline score, AdvaitaBench-N, averages two AI judges after correcting each judge's measured bias toward its own maker's models. 50 is the field average. Hover any point for detail.

Score vs words spent · does verbosity buy understanding?

Two judges, two stories · lenient grader vs strict grader

Each line is one model. The left column is an Anthropic judge with a permissive rubric, the right an OpenAI judge with a strict one. Lines that cross tell you the ranking depends on who grades, which is exactly why the headline score uses both.

Skill fingerprint · strict scores by family, dot area = score

How it works

Three design decisions do most of the work. Full methodology, rubrics, and the bias-correction formula are in the blog and the repo.

01Pressure, not trivia Multi-turn debates quote genuine scripture against the model until it holds or folds 02Reference-based grading Judges verify against written reference answers, with hard caps for known failure modes 03Judge bias, measured and corrected Each AI judge favored its own maker's models. We measured it and subtracted it

The questions

All 84 tasks are public. Filter by family. "Novel" tasks were written for this benchmark and exist nowhere in training data; "multi-turn" tasks include scripted pushback.