What the Einstein Test Actually Tests

A reading note on Philip Ball’s Nature feature about language models and creative breakthroughs

A note on what this is. This is a reading note, not a reprint. The article under discussion is Philip Ball, “The Einstein test: what happens when AI tries to rediscover relativity?”, Nature 657, 338–340 (9 September 2026), DOI 10.1038/d41586-026-02804-x. I read it in the Chinese edition published by the Nature Portfolio WeChat account 自然系列 on 10 November 2026, translated by Springer Nature’s Shanghai office. Where I quote, the wording is rendered back from that Chinese edition and may differ from the English original. All views and any errors below are mine.

Philip Ball’s feature is the kind of survey I rarely come across: nothing in it is padded, and nothing in it is selling anything. It takes a question that is usually asked rhetorically — will AI make a breakthrough like general relativity? — and turns it into a series of concrete, already-running experiments, then reports what those experiments actually produced. The experiments are small, some are unpublished blog posts, and the results are almost uniformly negative. That is exactly what makes the piece valuable.

My own view is blunt: only by being clear about what LLMs cannot do can we see what they are good for, and only then can we decide where human effort should go. Most of the discourse around AI for science fails at this first step. This note reconstructs the argument, marks the parts I find most consequential, and ends with the direction I think it points toward.

The test

In February, at the India AI Summit, Google DeepMind co-founder Denis Hassabis suggested training a language model on everything known up to a fixed cutoff — he proposed 1911 — and asking whether it can arrive at general relativity. His framing: this would be “a good test for AGI.”

The idea did not stay hypothetical. In December 2024, Owen Evans of the non-profit Truthful AI had already argued for “retro” or historical models trained only on data from before some date, precisely to ask what such a model might rediscover. This year several teams built them, including at least one attempt at Hassabis’s specific test.

A second strand is methodological rather than commemorative. Nick Levine, an independent computer scientist, and colleagues built a model restricted to knowledge available up to 1930 — a year chosen for a prosaic reason: works published then entered the US public domain in early 2026. The questions that model is supposed to answer are the ones that became askable right after its cutoff: the Turing machine, Gödel’s incompleteness theorems, the neutrino. A third strand, from a group at the University of Zurich, is a whole family of dated models — Ranke-4B, with cutoffs at 1913, 1929, 1933, 1939 and 1946, each a historically meaningful hinge — built to ask not whether a model can be a genius but whether it can produce ideas that carry traces of what came next. As the economist Daniel Göttlich, now at ETH Zurich, puts it: they are testing “the spark of genius,” not genius.

“LLMs can’t jump”

The most quoted objection in the piece belongs to DeepMind’s Tom Zahavy, whose January paper is titled, flatly, LLMs Can’t Jump. His argument is about the shape of the inference, not its accuracy. Progress of the Einstein kind requires abduction — the creative leap from particular phenomena to their causes — not induction from accumulated instances. Modern models do the second extremely well and have, as far as anyone can tell, no machinery for the first.

The failure mode is therefore not that models produce nonsense. It is that they produce the statistically likely answer to a well-posed prompt, which is a different thing from noticing that the prompt is the wrong one. Ball puts the point where it belongs, next to Thomas Kuhn: the imaginative leap tends to happen when there is very little data to fit, or when an anomaly refuses to fit the standard scenario. Both are conditions under which a system optimized for interpolation has nothing to optimize.

The constructive contrast is a familiar episode in the history of physics. Kepler’s meticulous observations of planetary motion, compressed into regularities, gave Newton the foothold for a law of gravitation and a mechanics of bodies under force. That is a world model built from sparse data: a small number of principles, from which an enormous range of behaviour follows. The natural question is whether a model can do the same thing in reverse — infer the principle, not just the fit.

What the experiments found

Sendhil Mullainathan at MIT tested exactly that. In a July conference paper, he and his colleagues trained a foundation model on synthetic data from many planetary systems, all obeying Newtonian mechanics. The model never recovered the actual law — the relation between force, mass and separation. It produced a different law for each planetary system, each wrong in a different way. This is worth sitting with. The model was faithful to its training signal and still missed the invariant, because the invariant is not what the training signal rewards.

Michael Hla, an independent researcher in San Francisco, ran the more romantic version: a model called Machina Mirabilis, trained only on pre-1900 data, asked in March whether it could reach quantum mechanics, special relativity or general relativity. He supplied hints — the photoelectric-effect observations in one case, the core of the elevator thought experiment in another. There were, he reports, “flashes of intuition”: facing the photoelectric effect, the model described light as breaking into discrete pulses, which is the photon waiting to be named. Mostly, though, it failed, and failed in a specific way — fluent, plausible-sounding physics with nothing underneath. Ball’s phrase for it is parroting; Hla’s diagnosis is the absence of any strong internal representation of the world to reason from.

Hla’s second finding is unglamorous and, for anyone who has built a training corpus, entirely expected: the historical record is a mess. Non-standard English and distortions introduced by digitizing printed matter make a date-constrained model much harder to train than it looks.

Levine’s group hit the complementary problem. Even restricting to 1930 does not work cleanly, because the material is a drop in the bucket and because dates leak. Asked about the 1950s, the model answers — and answers about the Roosevelt administration, which ran to 1945. When a corpus’s provenance is ambiguous or its dates are approximate, filtering everything after a cutoff becomes guesswork. Levine, a former quantitative forecaster in economics, is nevertheless interested in the forward-looking use: letting this “1930 mind” make predictions, including about things of the kind you see in prediction markets, and testing whether a model trained through January of a given year produces anything not yet known by June. His expectation is that it will, “purely because of the scale of data and resources involved.”

Mullainathan, in the same piece, is the one who names what is missing: such a model does not need a better corpus of knowledge so much as data about the real world — what the physical world is like — and a “mathematical space,” meaning how mathematical objects are manipulated. In his words, both are “still lagging in the current era.”

Two asymmetries

Two structural points in the feature seem to me more durable than any individual result.

The first is verification. Jacob Andreas at MIT observes that generating a theory like general relativity as one probabilistic output among many candidate theories is not, in itself, an obstacle — a language model can emit a thousand theories. The difficulty is having any basis for saying which of them is right, or even which is worth testing. The orbital-mechanics model is the demonstration: a thousand laws, no way to choose. In mathematics this bottleneck partly dissolves, because a proof can be checked step by step; that is why the Erdős-conjecture episode, where an OpenAI model helped a mathematician push an eighty-year-old problem forward, is a real advance and also a bounded one. It required some new abstract work, and it landed on a line of reasoning the literature already contained. It was not a bolt from the blue, and the checking was the thing that made it usable.

The second is significance. Andreas again: an LLM has no working notion of which mathematical theorem is interesting, and the capacity to tell what matters is, quite simply, a large part of what expertise is. This is the point I would underline hardest in the whole article. Prediction accuracy is measurable, so it gets optimized; significance is not measurable in the same way, so it does not. A system trained against a measurable objective will, over time, become excellent at the measurable thing and indifferent to everything else — which describes the current state of the field fairly well.

The outliers get averaged away

The consequence shows up in the sociology of discovery, not just its epistemology. Transformative shifts in science tend to come from attention to small discrepancies — the outlier that does not fit. Outliers are precisely what a model trained to minimize average error is built to smooth over. Ido Kaminer at the Technion makes the point through the psychology of the people who make such shifts: in physics, discovery has often depended on a persistent, almost obsessive attachment to an idea that everyone else regards as marginal, sustained for years.

His example is Max Planck, who would not accept that his blackbody formula merely matched the observed spectrum closely enough, and whose refusal to settle produced the 1900 quantum hypothesis. And, contrary to the popular story, Einstein was not much troubled by the failure of experiments to detect the aether. The empirical pressure behind special relativity was weaker than the textbooks suggest; what carried it was conceptual.

Ball’s general statement of this is the sentence I would keep: breakthroughs rarely fall out of the data. They come from a sustained and delicate interplay between theory, concepts, experiment and experimental technique — and in fields like molecular biology, which are not governed by a short list of fundamental laws, this is even more true. He gives two lovely examples of progress that came from reclassifying a nuisance as a feature: intrinsically disordered proteins, long thought unanalysable by crystallography because they have no stable structure, and biomolecular condensates, observed decades before anyone proposed the concept and ignored in the meantime.

What would have to change

Kaminer’s prescription is the natural conclusion: if the goal is a relativity-class discovery, do not optimize only for predictive accuracy. Reward parsimony and explanatory power as well. That is a statement about objective design, and it runs directly against the strongest empirical regularity in the field — what Rich Sutton calls the bitter lesson, that assumptions about how humans think, baked into a system, tend not to survive in the long run. What reliably improves performance is more compute and more data. Models are judged by being fast, accurate and useful, and need not reveal the process behind the phenomenon. AlphaFold is the canonical case: it predicts folded structures beautifully without answering how proteins fold.

Given that, Andreas’s forecast is the plausible one. Language models will, at least at first, make very limited discoveries, and they will make them in situations where we already know clearly where the important problem is, borrowing an existing solution or an analogy from elsewhere. A researcher may then read the result and think: I should have done a PhD in that field, and I would have found this. Nobody will think that about general relativity. Andreas’s imagined reaction to that is the right note to end on — not “why didn’t I think of this,” but “how on earth did he come up with it?”

My reading

What I take from the piece is a discipline of attention. The useful question about a model is never whether it is impressive; it is which part of the research loop it is actually closing, and what it would take to check the closure. Ball’s survey is unusually good at keeping that question open, and it does so by refusing to flatter either side.

The specific claims I find most persuasive:

  • Zahavy’s framing, that LLMs cannot jump, is the correct frame, because it locates the limitation in the kind of inference rather than in the amount of knowledge.
  • The gap is judgement, not information. A model that can distinguish what matters from what does not — the sentence I keep returning to — is where the real distance from human expertise lies.
  • Noticing the outlier is the function most at risk, since averaging is not a bug of these systems but the mechanism by which they work.
  • Mullainathan’s requirement — real-world data plus a “mathematical space” — is the right specification of the missing component, and it is missing on both counts at once.

Taken together these point, for me, toward a specific target rather than a general hope: a mathematical-physical world model. Not a larger corpus, and not a better predictor of the next token or the next benchmark, but a system that carries the structure of the physical world and the structure of mathematical objects together, so that manipulating one can be constrained by the other. Everything the feature documents is a measurement of the distance to that. Knowing the distance precisely is more useful than any claim that we are almost there.

A two-by-two map of LLM-driven discovery: guided search, the Einstein test, workhorses, and extrapolation

Where the cases in the feature land. The horizontal axis is the cost of verifying a result; the vertical axis is whether the required inference stays inside the training distribution. The blue and teal panels are where the reported wins live. The Einstein test sits in the upper right, the quadrant with no verifier.


中文导读

Philip Ball 在 Nature 上的这篇特写(2026 年 9 月 9 日,第 657 卷,338–340 页,DOI: 10.1038/d41586-026-02804-x),把一个常被当作修辞的问题——AI 能否做出广义相对论级别的突破——拆成了一组已经在跑的实验,然后如实报告了结果。这些实验规模都不大,有的只是个人博客,结论几乎没有例外地是否定的。这正是它的价值所在。

文章的主线有三段。其一是「爱因斯坦测试」本身:Hassabis 提议用截至 1911 年的知识训练一个模型,看它能否自己走到广义相对论;此后 Evans、Hla、Levine 以及苏黎世大学的 Ranke-4B 模型家族,各自用不同的截止年份做了尝试。其二是 Zahavy 的 LLMs Can’t Jump:问题不在于知识量,而在于这类进展需要的是溯因推理——从个别现象跳到原因——而不是从数据中归纳普遍规律。其三是失败的具体形态:Mullainathan 的轨道力学基础模型为每个行星系统推出一条不同的「引力定律」,每条错得都不一样;Hla 的 Machina Mirabilis 偶有「直觉的闪光」,多数时候只是在「鹦鹉学舌」;Levine 的 1930 模型会泄漏 1930 年之后的信息。

我认为最值得记住的是两处结构性论断。一是验证的不对称:模型可以生成一千个理论,却没有办法判断哪个是对的、哪个值得检验;数学领域因为可以逐步检查而缓解了这一困境,这也是 Erdős 猜想那次进展既真实又有边界的原因。二是重要性的判断:「模型辨别何者重要的能力,实际上正是与人类专家的一大差距。」科学认识上的变革性转变往往源于对细微差异的关注,而这类「离群值」恰是 AI 模型倾向于平均掉的。再加上 Sutton 的「苦涩教训」与 Kaminer「不能只为预测精度做优化,还要奖赏简洁性与解释力」的对照,文章实际上已经把能力边界画得很清楚了。

我自己的判断:这是我认为非常客观且全面的综述性博文,可以说是难得的没有任何太虚的东西,可谓道尽了 LLM 和 Agent 用于科学研究的本质和能力边界。这是非常重要的认识——只要知道 LLM 不能做什么,才能清楚 LLM 擅长做什么,更才能明确我们人类应该朝什么方向努力。其中 LLM 是否能 jump 的那篇文章我个人尤为推崇和喜爱;「模型辨别何者重要的能力,实际上正是与人类专家的一大差距」「科学认识上的变革性转变往往源于对细微差异的关注,这类『离群值』恰是 AI 模型倾向于平均掉的」「所需模型不仅要用知识语料库,还需要真实世界的数据(即物理世界是什么样的)以及『数学空间』(即如何操作数学对象)来训练……在当前时代尚属滞后」这几处,都是我非常喜欢并强烈认同的观点。它们与我推崇的一种「数理世界模型」的目标不谋而合。


Source

Ball, P. The Einstein test: what happens when AI tries to rediscover relativity? Nature 657, 338–340 (2026). doi:10.1038/d41586-026-02804-x

Chinese edition: 自然系列(Nature Portfolio 官方账号),2026 年 11 月 10 日,施普林格·自然上海办公室翻译。原文版权归 Springer Nature Limited 所有;本页为阅读笔记与评论,不是转载,引用文字为从中文版回译的短句,完整内容请阅读原文。