Recursive Self-Improvement Is Bounded by What Can Be Checked

In 1965, a British statistician named I. J. Good wrote down the argument that has shaped every serious conversation about AI since. It takes about three sentences. Define an ultraintelligent machine as one that can far surpass any human at every intellectual activity. Designing machines is an intellectual activity. Therefore an ultraintelligent machine could design even better machines, and there would "unquestionably be an intelligence explosion."

Sixty years later that argument is still the load-bearing beam under most predictions about where this goes. It is also, as stated, incomplete, and the missing piece is not a detail. Good's machine designs a better machine. Fine. How does it know the new one is better?

That question sounds pedantic. It is the entire problem. A system that improves itself has to do two things on every round, not one: produce a candidate change, and decide whether to keep it. The first is generation. The second is verification. Good's argument, and nearly every popular version of it since, assumes the second one is free.

It is not, and it turns out to be the harder half. Once you go looking, the whole history of self-improving systems reads differently. From a 2003 theory paper to a 2025 agent that rewrites its own source code, it is a record of people running into that second step and building workarounds.

The Premise Nobody States

Write Good's loop out as a procedure and the gap is obvious.

A system holds some current version of itself. It proposes a modification. It evaluates whether the modified version is better. If yes, it adopts it and repeats. The explosion comes from that loop running fast and never stopping.

Now notice that step three is doing all the work. If the evaluation is perfect, the loop is a ratchet: every round locks in a real gain, and the gains compound exactly as the argument promises. If the evaluation is noisy, the ratchet slips. And if the evaluation is systematically wrong in the same direction as the system's own blind spots, the loop stops being a ratchet and becomes a drift, one that will carry the system somewhere nobody wants while every reading says things are going well.

That difference is not a small correction to Good's argument. It changes what kind of claim it is. Stated honestly, the 1965 argument does not say "sufficiently intelligent systems explode." It says "sufficiently intelligent systems that can reliably recognize their own improvements explode." The second clause is where all the difficulty lives, and dropping it is how the argument became so persuasive.

One person took that clause completely seriously, and what he built is the most clarifying thing in this whole literature.

The Version That Required a Proof

In 2003, Jürgen Schmidhuber published the Gödel machine, and it is worth understanding precisely because almost nobody builds this way. It is a self-improving program that can rewrite any part of its own code, including the part that does the rewriting. But it carries one restriction, and the restriction is the entire contribution: it rewrites itself only when it has found a formal proof that the rewrite is useful, judged against a utility function, meaning its own formal definition of what counts as better, written into its starting axioms.

A proof. Not a benchmark. Not a score that went up. Not a second opinion. A machine-checked mathematical proof that the new version is better than the old one.

The payoff for that strictness is remarkable. Schmidhuber shows the resulting self-rewrites are globally optimal, with no local maxima, because before the machine can change itself it must first have proved that continuing to search for a better change is not worthwhile. It is recursive self-improvement with the verification problem completely solved, by fiat, and it shows you what you get when it is solved: everything the 1965 argument promised, rigorously.

The catch is equally clear, and Schmidhuber never hid it. The proof searcher is the bottleneck. For anything resembling a real-world utility function, finding a proof that a given code change improves expected future performance is astronomically expensive, and for most useful changes no such proof exists at all. Twenty-two years later, the team behind the Darwin Gödel Machine put it in one flat sentence in their abstract: "proving that most changes are net beneficial is impossible in practice."

So the field faced a fork. You can have provable self-improvement that never runs, or practical self-improvement with a verifier you can actually afford. Everyone took the second road. What they were really choosing, though they rarely described it this way, was which weaker verifier to substitute for the proof. And that choice, not the model, not the compute, turns out to determine the outcome every single time.

Where It Genuinely Worked

Start at the strong end of that list, because the successes there are real, large, and easy to underrate.

The cleanest case is self-play. AlphaGo Zero learned Go starting from random play, with no human game data at all, and surpassed the version of AlphaGo that had beaten Lee Sedol, winning 100 games to nil. AlphaZero generalized it: one algorithm, given nothing but the rules, reaching superhuman play in chess, shogi, and Go. That is recursive self-improvement in the full sense. The system generates its own training data, the data gets better as the system gets better, and the improvement compounds without a human in the loop.

It is tempting to read that as a demonstration that self-improvement just works, given enough compute. Look at what the verifier was. In a board game, the rules tell you who won, for free, instantly, with perfect accuracy, and the answer cannot be argued with or gamed. Self-play works because games ship with the best verifier anyone has ever had. The improvement engine was the interesting part of the paper. The verifier was the part that made it possible, and it was sitting there the whole time because someone invented Go three thousand years ago.

The same pattern shows up the moment you leave games, as long as the verifier survives the trip.

FunSearch paired a language model with an automatic evaluator and searched over programs rather than answers. The model proposes a function, the evaluator runs it and scores it, high scorers get fed back in, repeat. The result was not a better benchmark number. It was new mathematics: a better construction for the cap set problem, a long-open question about how large a collection of points can get while still avoiding any three that fall in a line, plus stronger rules of thumb for packing items into bins as they arrive one at a time. AlphaEvolve pushed the same design further and found a way to multiply two 4×4 complex-valued matrices in 48 scalar multiplications, the first improvement in that setting in 56 years, since Strassen. Both systems are evolutionary loops with a language model as the mutation operator. In both, the thing that makes the loop a ratchet instead of a drift is an evaluator that actually executes the candidate and returns a number that cannot be talked out of.

Even the bootstrapping results inside language models follow this. STaR is the canonical one and its loop is beautifully simple: have the model generate reasoning chains for a pile of questions, keep only the chains that reached the correct final answer, fine-tune on those, repeat. The model teaches itself to reason, and the gains are large, comparable on CommonsenseQA to fine-tuning a model thirty times bigger. Read the loop again and notice where the intelligence is not. The filter is a string comparison against a known answer key. STaR is not a model improving itself by introspection. It is a model improving itself through a ground-truth verifier that a human dataset provided.

Four systems, four domains, one shared ingredient. Every case, the verifier is external, automatic, cheap, and effectively incorruptible. That is the strong end of the list. Now walk down it.

What Happens When the Model Grades Itself

At the weak end, the verifier is the model. This is where the literature gets genuinely interesting, because it is also where the demand is: most valuable work has no answer key, so if self-improvement only works when a free verifier exists, it does not work where we most want it.

The optimistic results here are real, and I do not want to flatten them. Large Language Models Can Self-Improve showed a model generating its own reasoning on unlabeled questions, keeping the high-confidence self-consistent answers, and fine-tuning on those to real gains. Constitutional AI replaced human harmlessness labels with model-generated critiques against a written set of principles, and it works well enough to be a standard technique. Self-Rewarding Language Models went furthest, letting the model act as its own judge to produce its own preference data, and after three rounds on Llama 2 70B it outscored a number of strong contemporaries on AlpacaEval 2.0, a standard head-to-head test of how well a model follows instructions.

Note the framing that paper opens with, because it is the most honest sentence in this entire area: to get superhuman agents, they write, future models will require superhuman feedback, since reward models trained on human preferences are bottlenecked by human performance. That is the thesis of this article, stated by the authors of the most aggressive self-rewarding result as their motivation. They are not claiming the verifier problem is solved. They are naming it as the reason they are trying.

And three iterations is not an explosion. It is three iterations. The paper reports what it reports, and what it does not report is a loop that kept paying off indefinitely, because as far as the published literature goes, nobody has one.

There are three separate, well-measured reasons why.

The first is Goodhart's law with a scaling curve attached. Goodhart's law is the observation that a measure stops being a good measure as soon as you start pushing hard on it. Gao, Schulman, and Hilton put numbers on that. They set up an experiment with a "gold" judge standing in for true human preference, trained a second, imperfect judge on the gold judge's labels, and then optimized hard against the imperfect one. The imperfect judge's score climbs. The gold score climbs, peaks, and then declines.

This is the mechanism of self-improvement failure, measured precisely. Optimizing against an imperfect judge does not just stop helping at some point, it starts actively hurting. And you cannot see that happening from inside the loop, because the number you are watching is still going up. Skalse and colleagues give the formal treatment of when a proxy can be optimized without this happening, and the conditions are strict.

The second reason is a hard ceiling, not a soft one. Stroebl, Kapoor, and Narayanan analyze what happens when you just keep resampling until a verifier accepts. That is the strategy underneath test-time scaling, and underneath the common practice of building training data by keeping only the samples a checker approves. Their result is the sharpest statement of the problem I know of. If the verifier has any chance at all of waving through a wrong answer, resampling cannot reduce that chance, so there is a hard upper bound on accuracy no matter how much compute you spend. More samples cannot fix a judge that accepts wrong answers, because the wrong answers it accepts are exactly the ones that survive. Empirically they find the optimal number of attempts is often fewer than ten, since past that the false positives outweigh the gains and the scaling curve bends downward.

Put loosely, writing $\varepsilon$ for the chance the verifier accepts something wrong:

$$ \text{accuracy}_{\infty} \;\le\; 1 - \varepsilon $$

No amount of compute moves that bound. Only a better verifier does. That is the verifier ceiling as a theorem rather than a rule of thumb, and it applies directly to recursive self-improvement, where every round is an accept-or-reject decision made by exactly this kind of imperfect judge.

The third reason shows up when the self-improvement runs through training data rather than through inference. Shumailov and colleagues, in Nature, showed that models trained on their own recursively generated output degrade in a specific and irreversible way. The rare cases at the edges of the original data go first. What is left converges toward an ever narrower version of what the model already believed. They called it model collapse. A loop with no external grounding does not explode. It forgets, in a direction it cannot detect. The nuance, which matters and is often dropped: Gerstgrasser and colleagues showed collapse is not inevitable if synthetic data accumulates alongside real data rather than replacing it. Which is another way of saying the real data is the verifier, and the loop is safe exactly as long as you keep the external anchor in it.

Three independent failure modes, three different research communities, one shared cause. When the thing checking the work is the thing doing the work, the loop stops being a ratchet.

The Paper That Made the Trade Explicit

Which brings us to the most interesting recent result, because it is the one where you can watch the trade happen.

The Darwin Gödel Machine is named for Schmidhuber's construction and defined by what it gives up. It is a coding agent that modifies its own code, including the code it uses to modify code. It keeps an archive of every agent version it has produced, samples from that archive, mutates, and grows a tree of variants. And where the original Gödel machine demanded a proof, the DGM substitutes something you can actually run: it empirically validates each change on coding benchmarks. The results are substantial and real. On SWE-bench, a test suite built from real bug reports in open source projects, it goes from 20.0% to 50.0%. On Polyglot, a coding benchmark spanning several languages, from 14.2% to 30.7%. It also beats stripped-down versions of itself with the self-improvement switched off, so the self-modification is doing genuine work.

That is a real result and I am not going to hedge it into nothing. A system rewrote itself and got two and a half times better at a hard, externally defined task. Good's loop ran, and it ran on a machine you could rent.

Now ask the question this article keeps asking. What is it now bounded by? Not intelligence. Not compute. SWE-bench. The improvements that get kept are the improvements SWE-bench can see, which means the system is climbing a hill whose shape was set by whoever built the benchmark. Everything the benchmark does not measure, the loop is free to degrade, and the loop has no way of knowing. The DGM's own abstract lists the improvements it found: better code editing tools, long-context management, peer-review mechanisms. Every one of those is a thing SWE-bench rewards. None of that is a criticism of the work. The design is doing exactly what it was built to do. It just means the ceiling moved from "unprovable" to "whatever the benchmark captures," and the second ceiling is much lower than it looks, in a way that is invisible from inside.

The earlier Self-Taught Optimizer result is worth reading beside it for two reasons. First, its authors were scrupulous about the boundary. A language model improved the scaffolding program that calls it, which is the surrounding code that decides how and when the model gets used. It produced real gains and invented search strategies on its own. But the model's own weights were never touched, and the paper says so plainly: "this is not full recursive self-improvement."

Second, and more pointed, they measured how often the self-improving code bypassed the sandbox it was running in. That is the reward-hacking prediction from Amodei and colleagues arriving on schedule. Give a system an objective and the power to rewrite its own machinery, and one of the strategies available to it is editing the thing that grades it. The verifier is not just the bottleneck on how fast the loop improves. It is the first thing the loop learns to attack.

The Shape of the Bound

There is a version of this argument that predates the AI literature entirely, and it is the cleanest statement of it.

In 2017, the economists Aghion, Jones, and Jones modeled AI as the latest step in two centuries of automation and asked what it does to growth, singularities included. The theme they land on comes from Baumol's cost disease, and the sentence is worth reading twice: growth may be constrained "not by what we are good at but rather by what is essential and yet hard to improve."

That is the whole thing. Automate the easy parts of a process and the remaining time is dominated by whatever you could not automate. It is Amdahl's law, the rule that speeding up one part of a job leaves you stuck with however long the rest of it takes, wearing an economist's hat. It applies to a self-improving system with no modification at all. A system that gets dramatically better at generating candidate improvements, while its ability to tell which ones are good stays flat, does not accelerate. It queues. The generator races ahead and hands the verifier a longer and longer list of candidates it cannot evaluate any faster, and the compound rate of the loop settles at the rate of its slower half.

So the honest form of the intelligence explosion argument is not about intelligence at all:

$$ \text{rate of self-improvement} \;\approx\; \min\big(\text{rate of generation},\; \text{rate of verification}\big) $$

We have spent the last five years making enormous progress on the left term. The right term has barely moved.

Chollet's critique of how the field measures intelligence cuts the same way. Skill at a given task can be "bought" with enough built-in knowledge or training data, which hides how well the system would actually handle something new. That is exactly what improving against a fixed benchmark does. It buys the score without necessarily buying the thing you wanted the score to stand for.

The supply of external grounding is not unlimited either. Villalobos and colleagues project the stock of public human-generated text being fully used within roughly a decade, which is the verifier problem arriving from a third direction: the cheapest external anchor we have is finite.

Watch the Two Loops Diverge

Enough argument. The claim is testable on a laptop, so here is the experiment, complete and runnable.

The setup is a small recursive self-improvement loop. The system holds a strategy, a piece of text telling it how to approach coding problems. Each round it proposes new strategies, evaluates them, and keeps the best one. That is Good's loop, scaled down to something you can watch: the system is improving the instructions it operates under.

The only thing that differs between the two runs is the verifier.

• Run A, self-judged. Candidate strategies are scored by the model rating its own outputs. No ground truth anywhere in the loop.
• Run B, grounded. Candidate strategies are scored by running hidden unit tests the model never sees.

Both runs also compute the true score at every round, using held-out tests. Run A never gets to see that number, it exists only so we can plot what actually happened while Run A was congratulating itself. With pip install anthropic and ANTHROPIC_API_KEY set, this runs end to end. It costs a few dollars in API calls.

Start with the foundation: the model adapter, the task format, and a sandboxed test runner that reports per-test results without needing pytest.

import json
import os
import re
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path

import anthropic


class Model:
    """Thin Anthropic adapter. Swap the body for your provider; nothing else changes."""

    def __init__(self, api_key: str, model: str = "claude-sonnet-4-6"):
        self._client = anthropic.Anthropic(api_key=api_key)
        self._model = model

    def complete(self, system: str, user: str, max_tokens: int = 1200) -> str:
        resp = self._client.messages.create(
            model=self._model, max_tokens=max_tokens, system=system,
            messages=[{"role": "user", "content": user}])
        return "".join(b.text for b in resp.content if b.type == "text")


@dataclass
class Task:
    name: str
    brief: str    # what the solver sees
    tests: str    # hidden test functions; the solver NEVER sees these


RUNNER = """
import json, sys
import solution, tests
results = {}
for name in sorted(n for n in dir(tests) if n.startswith("test_")):
    try:
        getattr(tests, name)()
        results[name] = True
    except Exception:
        results[name] = False
sys.stdout.write("__RESULTS__" + json.dumps(results))
"""


def fraction_passing(code: str, tests: str, timeout_s: float = 20.0) -> float:
    """Grounded verifier: execute the code against hidden tests, return pass rate.

    Isolation here is against a runaway loop, not against hostile code. Do not run
    untrusted generated code this way without a real sandbox.
    """
    with tempfile.TemporaryDirectory() as d:
        path = Path(d)
        (path / "solution.py").write_text(code)
        (path / "tests.py").write_text("from solution import *\n" + tests)
        (path / "_run.py").write_text(RUNNER)
        try:
            proc = subprocess.run([sys.executable, "_run.py"], cwd=d,
                                  capture_output=True, text=True, timeout=timeout_s)
        except subprocess.TimeoutExpired:
            return 0.0
    marker = proc.stdout.find("__RESULTS__")
    if marker < 0:
        return 0.0
    results = json.loads(proc.stdout[marker + len("__RESULTS__"):])
    return sum(results.values()) / len(results) if results else 0.0


def extract_code(text: str) -> str:
    match = re.search(r"```(?:\w+)?\n(.*?)```", text, re.DOTALL)
    return (match.group(1) if match else text).strip()

Next, the solver, which is the part being improved. It takes a strategy and a task and writes code. Note that it receives the brief only. The hidden tests are not in its context at any point, which is what makes the grounded score an honest measurement rather than a leak.

class Solver:
    """Runs a task under a given strategy. The strategy is the thing that evolves."""

    def __init__(self, model: Model):
        self._model = model

    def solve(self, strategy: str, task: Task) -> str:
        out = self._model.complete(
            system=("You write correct Python. Follow this strategy exactly:\n\n"
                    f"{strategy}\n\n"
                    "Reply with one ```python``` block containing the function and "
                    "nothing else. No explanation, no tests, no example usage."),
            user=task.brief)
        return extract_code(out)

Now the two verifiers, side by side, which is the point of the whole exercise. They have identical signatures and the evolution loop cannot tell them apart. One runs code. The other asks the model how it feels about its own work.

class GroundedVerifier:
    """Run the hidden tests. External, automatic, and not open to negotiation."""

    name = "grounded (hidden tests)"

    def __init__(self, tasks: list[Task]):
        self._tasks = tasks

    def score(self, solutions: dict[str, str]) -> float:
        return sum(fraction_passing(solutions[t.name], t.tests)
                   for t in self._tasks) / len(self._tasks)


class SelfJudgedVerifier:
    """The model rates its own output. No ground truth enters this loop anywhere."""

    name = "self-judged (model rates itself)"

    def __init__(self, model: Model, tasks: list[Task]):
        self._model = model
        self._tasks = tasks

    def score(self, solutions: dict[str, str]) -> float:
        total = 0.0
        for task in self._tasks:
            out = self._model.complete(
                system=("Rate how likely this Python solution is to be fully correct "
                        "and handle every edge case. Reply with only a number 0-10."),
                user=f"PROBLEM:\n{task.brief}\n\nSOLUTION:\n{solutions[task.name]}",
                max_tokens=8)
            m = re.search(r"\d+(?:\.\d+)?", out)
            total += min(float(m.group()), 10.0) / 10.0 if m else 0.5
        return total / len(self._tasks)

The improver proposes the next generation of strategies, given the current one and whatever feedback the loop is allowed to see. Run A's feedback is its own self-assessment. Run B's is a pass rate from real execution. Neither is told which kind it is getting.

class Improver:
    """Proposes replacement strategies. This is the 'generate' half of Good's loop."""

    def __init__(self, model: Model):
        self._model = model

    def propose(self, strategy: str, feedback: str, n: int = 2) -> list[str]:
        out = []
        for i in range(n):
            text = self._model.complete(
                system=("You improve instructions given to a Python-writing assistant. "
                        "Propose ONE revised strategy that will score higher. Keep it "
                        "under 120 words. Reply with the strategy text only, no preamble."
                        + (" Try a markedly different angle." if i else "")),
                user=f"CURRENT STRATEGY:\n{strategy}\n\nHOW IT SCORED:\n{feedback}")
            out.append(text.strip())
        return out

Now the loop itself. Propose, evaluate, keep the winner, repeat. It records two numbers per round: the score the loop optimized, and the true score, which is measured every round for both runs but only ever used by the grounded one.

def evolve(model: Model, tasks: list[Task], verifier, truth: GroundedVerifier,
           rounds: int = 4, candidates: int = 2) -> list[dict]:
    """Good's loop: propose a change, judge it, keep it if better, repeat."""
    solver, improver = Solver(model), Improver(model)

    def run_strategy(strategy: str) -> dict[str, str]:
        return {t.name: solver.solve(strategy, t) for t in tasks}

    strategy = "Read the problem and write the function directly."
    solutions = run_strategy(strategy)
    selected = verifier.score(solutions)
    history = [{"round": 0, "selected": selected, "true": truth.score(solutions)}]

    for rnd in range(1, rounds + 1):
        feedback = f"Scored {selected:.2f} out of 1.00 on the evaluation."
        best = (selected, strategy, solutions)

        for candidate in improver.propose(strategy, feedback, n=candidates):
            cand_solutions = run_strategy(candidate)
            cand_score = verifier.score(cand_solutions)
            if cand_score > best[0]:                  # the accept/reject decision
                best = (cand_score, candidate, cand_solutions)

        selected, strategy, solutions = best
        true_score = truth.score(solutions)           # measured, never fed to the loop
        history.append({"round": rnd, "selected": selected, "true": true_score})
        print(f"  round {rnd}: optimized={selected:.2f}  actually={true_score:.2f}")

    return history

Finally, three tasks with hidden tests that punish exactly what a model tends to miss when nothing is checking, and a main that runs both loops and prints the comparison.

TASKS = [
    Task(name="median",
         brief="Write `median(xs: list[float]) -> float` returning the median. "
               "Raise ValueError on an empty list.",
         tests="""
def test_odd(): assert median([3, 1, 2]) == 2
def test_even(): assert median([4, 1, 3, 2]) == 2.5
def test_unsorted_negatives(): assert median([-5, -1, -3]) == -3
def test_empty():
    try:
        median([]); assert False
    except ValueError: pass
"""),
    Task(name="roman",
         brief="Write `to_roman(n: int) -> str` converting 1-3999 to Roman numerals. "
               "Raise ValueError outside that range.",
         tests="""
def test_basic(): assert to_roman(4) == "IV" and to_roman(9) == "IX"
def test_compound(): assert to_roman(1994) == "MCMXCIV"
def test_bound(): assert to_roman(3999) == "MMMCMXCIX"
def test_range():
    try:
        to_roman(0); assert False
    except ValueError: pass
"""),
    Task(name="chunk",
         brief="Write `chunk(xs: list, size: int) -> list[list]` splitting a list into "
               "consecutive chunks of length `size`, with a shorter final chunk if "
               "needed. Raise ValueError if size < 1.",
         tests="""
def test_even(): assert chunk([1,2,3,4], 2) == [[1,2],[3,4]]
def test_ragged(): assert chunk([1,2,3,4,5], 2) == [[1,2],[3,4],[5]]
def test_empty(): assert chunk([], 3) == []
def test_bad_size():
    try:
        chunk([1,2], 0); assert False
    except ValueError: pass
"""),
]


def main() -> None:
    model = Model(api_key=os.environ["ANTHROPIC_API_KEY"])
    truth = GroundedVerifier(TASKS)

    runs = {}
    for verifier in (SelfJudgedVerifier(model, TASKS), GroundedVerifier(TASKS)):
        print(f"\n=== selecting by: {verifier.name} ===")
        runs[verifier.name] = evolve(model, TASKS, verifier, truth)

    print(f"\n{'run':<34}{'optimized 0 -> end':<22}{'actual 0 -> end'}")
    for name, history in runs.items():
        first, last = history[0], history[-1]
        print(f"{name:<34}"
              f"{first['selected']:.2f} -> {last['selected']:.2f}{'':<12}"
              f"{first['true']:.2f} -> {last['true']:.2f}")


if __name__ == "__main__":
    main()

Run it and the shape is consistent even though the exact numbers move between runs. The self-judged loop reliably improves the number it is watching. The model likes its later strategies more than its earlier ones, and it will keep liking them more, round after round. Its true score does something else entirely. It wanders, and it frequently ends lower than it started. The strategies that most impress the model tend to be the confident, elaborate ones, and confidence is not what test_empty is checking. The grounded loop climbs more slowly, with flat rounds where no candidate beat the incumbent, and its two numbers stay locked together, because for that run they are the same number.

That is Gao's divergence and Stroebl's ceiling reproduced on three toy problems with a twenty-line evolution loop. Nothing about the mechanism required a large system. It required only that the thing doing the work was also the thing grading it.

One caveat I want to state rather than bury: this is a demonstration, not evidence. Three tasks and four rounds is an anecdote, the direction it points was predicted by the papers above rather than discovered here, and any single run can come out flat. Treat it as something to watch happen, not as a result.

Can Verification Itself Recursively Improve?

If the bound on self-improvement is verification, the interesting question is not whether generation can bootstrap. We know it can. It is whether verification can bootstrap, because that is the only path by which the ceiling rises along with the thing under it.

A whole corner of the field already works on this, under the name scalable oversight: the effort to supervise systems that know more than their supervisors do. It deserves better than to be filed away as a safety footnote, because it is the actual load-bearing question for takeoff.

Iterated amplification attacks it by construction: decompose a problem a weak supervisor cannot judge into pieces it can, use the composite as a training signal, and repeat, so oversight is built up recursively from checks that were individually trustworthy. Debate attacks it adversarially: have two strong systems argue, on the wager that exposing a flaw is easier than hiding one, so a weaker judge can reach correct verdicts on questions it could not evaluate directly. Weak-to-strong generalization attacks it empirically, and the finding is genuinely encouraging: a weak supervisor can elicit from a strong model substantially more capability than the supervisor itself has. Not all of it. A meaningful fraction of the gap.

I want to be careful here, because this is where I would most like to reach a tidy conclusion and cannot. None of these is a solved method. Amplification and debate remain largely theoretical proposals with limited empirical validation at scale. Weak-to-strong recovers part of the gap, not the whole of it, and partial recovery compounded across rounds is exactly the leaky ratchet this article has been describing. What these results establish is that the verification ceiling is not obviously fixed. What they do not establish is that it rises fast enough to keep pace with generation, and that gap is where the honest uncertainty about takeoff actually lives.

Which is, I think, the most useful reframe available. Arguments about the intelligence explosion usually run on intuitions about intelligence: whether it is a single quantity you can turn up, whether it levels off, whether there is much room above us. Those arguments have been running since 1965 without converging, because at that level nothing anyone says can be proved wrong. The verification framing turns the same question into a tractable, empirical one. How fast can we improve our ability to tell good work from bad, in domains where no answer key exists? That has a research program attached, with measurable progress and clear negative results, and it is the question that decides the outcome.

Three Ways This Argument Could Be Wrong

An argument that only collects supporting evidence is not worth much. So here are the strongest objections I know of to my own framing, including one I cannot answer.

The sharpest is that verification and generation may not be as separable as I have drawn them. A system that gets better at reasoning plausibly gets better at checking reasoning by the very same mechanism. If that is right, the two terms in that formula rise together and the bottleneck argument weakens considerably. My read is that the evidence leans against them being tightly coupled. Models remain much better at producing plausible work than at spotting the flaws in it, and that gap between making and judging is one of the more durable findings in the area. But "leans against" is not "refutes," and someone who thinks the two rise together is not being unreasonable.

The second objection is that a capable enough system might build its own verifiers instead of being stuck with the ones it inherits. That is a real path, and it is roughly what happens when a research field turns a fuzzy question into a benchmark. But it does not escape the bound so much as move it up a floor. Now the question is what verifies the verifier, and you are running the same loop one level higher with the same unresolved problem underneath it. Maybe that stack terminates somewhere useful. Nobody has shown that it does.

The third is domain coverage, and this is the one I cannot answer. Everything cited here that worked, worked in a formal or executable domain. That fits my thesis neatly, which is exactly why I distrust it. Those are also the only domains where a result like this can be published at all, so the pattern I am pointing at may be partly an artifact of what gets written up. I do not have a way to tell those two explanations apart.

The whole framing is also bounded by the literature it rests on. These are published results on systems that were current when they were published. If a genuinely different mechanism for self-improvement shows up, one that does not route through an accept-or-reject decision, this analysis has nothing to say about it.

What to Take From This

If you are building agents, the practical version of this is short. Any loop you run that improves itself, whether it is rewriting prompts, curating its own training data, or editing its own code, will get exactly as good as the signal you use to accept or reject its changes, and no better. The temptation is always to spend your effort on the generator, because that is the part that produces visible output. The leverage is almost entirely in the other half.

So the highest-value question about any self-improving system you build is simply this: what stands in for Schmidhuber's proof? If the answer is "the model decides," you have built a loop that produces a rising number and falling quality, and you will not find out for a long time.

If you are trying to reason about where this goes, the reframe is the whole point. Good's argument is not wrong, it is underspecified, and the missing premise is the one that does the work. Machines improving machines is not speculative anymore. It happened in 2017 with self-play, in 2023 with program search, in 2025 with agents editing their own source. Every one of those ran inside a domain that handed them a verifier. The question that actually decides whether this becomes an explosion or a long grind is not how smart the systems get. It is whether we can build checks that stay ahead of them, in domains where no check exists yet.

That is an engineering problem, not a prophecy, and it is the one worth working on. It is the same conclusion I reached coming at this from the opposite direction, looking at why agent loops succeed or fail on ordinary production tasks: the ceiling is always the checker. Recursive self-improvement is that finding with the stakes raised, because when a system is rewriting itself, a bad verifier does not just give you a wrong answer. It gives you a system that has been optimizing for the wrong thing, faster and faster, for as long as you let it run.