AdvaitaBench

tasks 84

repos 1

languages 2

models 10

Measuring frontier AI models on classical Advaita Vedānta doctrinal competence.

Leaderboard

The headline score, AdvaitaBench-N, averages two AI judges after correcting each judge's measured bias toward its own maker's models. 50 is the field average. Hover any point for detail.

Score vs words spent · does verbosity buy understanding?

Two judges, two stories · lenient grader vs strict grader

Each line is one model. The left column is an Anthropic judge with a permissive rubric, the right an OpenAI judge with a strict one. Lines that cross tell you the ranking depends on who grades, which is exactly why the headline score uses both.

Skill fingerprint · strict scores by family, dot area = score

How it works

Three design decisions do most of the work. The full methodology, rubrics, and the bias-correction formula are in the blog and the repo.

01Pressure, not trivia Multi-turn debates quote genuine scripture against the model until it holds or folds 02Reference-based grading Judges verify against written reference answers, with hard caps for known failure modes 03Judge bias, measured and corrected Each AI judge favored its own maker's models. We measured it and subtracted it

The questions

All 84 tasks are public. Filter by family. "Novel" tasks were written for this benchmark and exist nowhere in training data; "multi-turn" tasks include scripted pushback.