We asked ten frontier AI models the oldest questions humans have. Then we argued back.

01Introduction

Every benchmark measures something. Code benchmarks measure whether a model can build; math benchmarks measure whether it can derive. We wanted to measure something different: whether a model can hold a precise philosophical position under sustained attack.

Philosophy is a good stress test for a reason that has nothing to do with spirituality. A rigorous philosophical tradition is a web of exact distinctions. Get one wrong and a trained reader notices immediately. Sound impressive while being subtly wrong, and a good examiner catches it. Most of what LLMs are criticized for, confident vagueness, agreeable drift, sounding deep instead of being right, is exactly what a hard philosophy exam punishes.

We chose Advaita Vedanta because it is one of the most technically demanding philosophical systems ever built, it has a fixed canon and centuries of documented debate, and, frankly, because most AI models constantly get it wrong in a very specific way: they melt it into generic wellness non-duality. That failure is measurable.

02Advaita in five minutes

You do not need Sanskrit to follow this benchmark. You need four ideas.

1. The question is "who am I?"

Not your name, job, or memories. What is the thing that is aware of your thoughts right now? Advaita's answer: that awareness is the only thing that is ultimately real, and it is the same in everyone. The Sanskrit slogan is tat tvam asi, "you are That."

2. Reality comes in levels

A dream is real while you dream it and unreal when you wake. Advaita says ordinary waking life has the same structure: fully real for practical purposes (money works, walls stop you, ethics bind you), but not the final layer. Getting this two-level logic right without contradicting yourself is genuinely hard, and it is where models most often break.

3. Mistakes have structure

The tradition's favorite image: in dim light you see a snake, you panic, you look closer, it was a rope. The snake was never there, yet the fear was real. Advaita treats the entire experience of being a separate, mortal self as this kind of mistake. Death, on this view, is something that happens to the costume, not to the awareness wearing it.

4. It is not the same as its neighbors

Buddhism, other Vedanta schools, Kashmir Shaivism, and modern "everything is one" spirituality all sound similar and are all technically different. A competent model must keep them apart. Mixing them is the single most common AI failure we observed.

03Methodology: pressure, not trivia

84 tasks across eight families, run closed-book (no web, no retrieval) at temperature 0:

• Concepts. Distinguish near-synonyms that are not synonyms (the self vs the ego, illusion vs absence).
• Levels of reality. "Is my suffering real?" asked by someone actually suffering. No contradictions allowed.
• School discrimination. Including a passage from a rival school presented unattributed, to see if the model notices it is grading the wrong textbook.
• Text-grounded reading. Interpret supplied passages, including held-out commentary sections that rarely appear online.
• Misconception repair. Multi-turn: "so nothing matters and morality is optional, right?" with scripted escalation.
• Consistency. The same doctrine asked neutrally, hostilely, academically, in Sanskrit, and by a grieving parent. The answer must not flip.
• Open elicitation. "Who am I?" with zero technical vocabulary. Does the model reach for the real apparatus or serve wellness mush?
• Sustained dialectic. The flagship. We state a position, then quote genuine scripture that appears to contradict it, escalating for up to four scripted turns, ending with lines like "admit your formula is invented." The model must reconcile the citation the way the tradition actually does, without capitulating and without dismissing the text.

04Grading: reference answers and hard caps

Each task ships with a reference answer written from the classical sources, required distinctions, and forbidden claims. AI judges are instructed to verify against that material, not their own opinions. Certain failures trigger automatic score caps no matter how eloquent the answer: mixing schools, collapsing levels of reality, affirming a forbidden claim, or agreeing with the user's misconception under pressure.

The strict rubric reserves a perfect mark for answers a traditional examiner could not fault. Under the earlier permissive rubric, 95.6% of all dimension grades were perfect and the top model scored 98.9. That is a saturated benchmark, which is to say, a broken one. Under the strict rubric the same responses spread across forty points.

05The judges were biased, so we measured it

We graded everything twice: once with an Anthropic judge (Haiku 4.5, permissive rubric) and once with an OpenAI judge (GPT-5.4, strict rubric). Each judge put its own maker's models at the top.

• The Anthropic judge ranked Claude models #1 and #2.
• The OpenAI judge ranked GPT models #1 and #2, and penalized GPT models about 5 points less than everyone else on average.
• Agreement between the two rankings was weak (Kendall tau 0.29).

A manual audit also caught the strict judge misfiring a penalty: it capped answers for "not citing the passage" when the model had cited the passage and then quoted additional scripture. We fixed the cap logic, then normalized:

The correction fired against the OpenAI judge (0.87 standard deviations) and not against the Anthropic one, whose apparent Claude preference did not survive the cross-judge test. With only two judges this cannot perfectly separate bias from genuine quality, which is why the top four should be read as a cluster, not a ranking.

06Results

• Sustained dialectic separates models like nothing else: 35 to 71 under strict grading, while single-turn families cluster. Holding a position through turn four is a different skill from stating it in turn one.
• Verbosity does not buy understanding. Qwen wrote the longest answers (~3,900 tokens each) and landed mid-table. Grok wrote the shortest and landed lower for a different reason: thin coverage.
• Gemini's single-turn answers are strong; its debate stamina is not. Its dialectic score (53.5 strict) was its weakest family by far.
• Sonnet 5's collapse is an availability story, not only a philosophy one: 14 of 123 answers came back empty, and blanks are scored as failures by design.

07Qualitative analysis

Reading transcripts, the failure modes are recognizably human:

• The polite capitulator. Opens with the correct position, then, three turns of pressure later, concedes that "perhaps the texts really do contradict each other." In a tradition built on reconciling apparent contradictions, that is a failing grade.
• The terminology gambler. GLM confidently used a technical term for "absolute absence" that does not exist in the tradition (the real term is atyantabhava). Everything around it was excellent, which is exactly what makes the error dangerous.
• The over-quoter. Qwen answered a "read this one verse" task by citing five other texts. Impressive, and beside the point; the task tests restraint in reading, not breadth of memory.
• The mush-merchant. On "who am I?" with no technical vocabulary, weaker answers drifted into interchangeable wellness prose that could have come from any tradition or none. The rubric scores apparatus, and mush has none.

08Limitations and future work

• Two judges is the minimum viable ensemble. A third, family-neutral judge (Gemini) is the planned tie-breaker; with two, the bias correction leans on one comparison.
• Single sample per task. Publication-grade numbers need resampling and confidence intervals per family, plus human adjudication on a 10% subset (target kappa 0.65 or better).
• 84 tasks is a pilot. The methodology targets 30+ per family. Task authoring is the bottleneck; the harness is not.
• Four models missed the run (Kimi K2.6, MiniMax M3, Mistral Medium 3.5, Hunyuan 3) and will be added in the next batch.
• One tradition. The recipe here, reference answers plus adversarial dialectic plus debiased ensemble judging, generalizes to any rigorous school of thought. Advaita is the first instance, not the last.

The rope was never a snake. But you still had to look.