Learning from Boris Cherny's live session

I watched the live coding session with Boris Cherny (Head of Claude Code at Anthropic) and Jared Sumner (creator of Bun). I wanted to get more than the surface, so I asked Claude and Codex to analyze the talk separately, then combined what they each found. What follows is the combined analysis, including Codex’s timeline and key ideas at the end.

Watch the original video on YouTube.

---

From “write the code” to “build the loop”

The talk is a live demo of how the Bun team runs Claude Code at scale through their bot RoboBun. The core lesson is that the engineer’s job has moved from writing all the code to building the loop around it. Issue, reproduce, fix, test, review, CI, confidence, merge.

Contents

1. What RoboBun actually is
2. The moving bottleneck and merge confidence
3. The merge confidence stack
4. Adversarial review and who fixes comments
5. CLAUDE.md as operational memory
6. Self-verification
7. Hill climbing
8. Tooling worth flagging
9. Generalizing beyond Bun
10. Action plan
11. Codex timeline and key ideas

1. What RoboBun actually is

RoboBun is a custom bot the Bun team built themselves. It is not an installable Claude Code feature. It is a GitHub webhook plus the Claude Code SDK plus their CI, wired together. The name is just a pet name.

On every new GitHub issue it does the following automatically. It spins up a container. It tries to reproduce the issue. It writes a failing test that captures the bug. It writes a fix. It verifies that the test fails on main and passes on the branch, and that verification is a hard gate. Without it, no PR opens. It opens the PR, then loops with review bots until the comments are resolved.

Over the last three months, RoboBun is the top contributor to Bun, ahead of Jared himself. Most of its PRs do not get merged.

Why a fresh container per issue

Issues should not see each other’s files or state. Repros often need a clean install. Fifty issues overnight means fifty containers in parallel with no conflicts. Failed repros get thrown out without cleanup. The container is the repro environment a maintainer would otherwise have built by hand.

2. The moving bottleneck and merge confidence

The principle is simple. Do not optimize what is already easy. Each time a bottleneck is solved, a new one appears, and that new one is where the next work belongs.

Era	Bottleneck	Solved by
Past	Getting working code	Better models
Recent	Knowing if code is correct	Self-verification (tests, CI access)
Now	Merge confidence. What proof do I need?	Unsolved (section 3)
Next	Planning and taste. What should we build?	Still human-heavy

3. The merge confidence stack

The actual answer to “what proof do I need” is a checklist of evidence. Five layers. If all of them are green, merge. If any are missing, investigate. Layers one through four map to what the video describes. Layer five is something Claude added, marked clearly below.

Layer 1. Test proof

A test captures the new behavior or the bug. It fails on main, which proves it is testing the actual thing and not just always passing. It passes on the branch, which proves the fix works. It lives in the right place and uses real dependencies rather than mocks of the system under test. Without this layer, you never merge, no matter how clean the diff looks.

Layer 2. CI proof

Full suite green on the branch. Lint, typecheck, build all green. Green on the first run with no flaky retries. Coverage did not drop on the changed files.

Layer 3. Adversarial review proof

Two independent reviewers, ideally from different vendors, have run. Every comment is resolved with a fix commit or dismissed with a reason. Nobody is still mid-conversation.

Layer 4. Behavioral proof

The change has to have actually run somewhere. Endpoint called with curl, button clicked in a browser, query executed against a real database. The observed output matched what was expected. For UI work that means a screenshot of the new state exists. For data work, the row in the database got inspected after the change. Layers one through three verify code. Layer four verifies behavior. This is what self-verification really means, and the gap here is what lets “tests pass but production is broken” happen. Most people skip this layer.

Layer 5. Blast radius proof (Claude addition, not in video)

The diff scope matches the issue. No scope creep. No unrelated files touched. No new dependencies without justification. A rollback path exists, ideally a single revert with no irreversible migrations.

The decision rule

All 5 layers green                              -> merge
Layer 4 missing but 1,2,3,5 green AND trivial   -> merge
Layer 1 or 2 missing                            -> never merge
Layer 3 missing                                 -> don't merge unless you read every line
Layer 5 sketchy                                 -> don't merge (scope creep hides bad PRs)

Bun has no auto-merge. A human presses the button every time. The bot just turns a thirty-minute review into a five-minute one. The bar shifted from “is this code passable?” to “do I trust the verification enough?“

4. Adversarial review and who fixes comments

Why CodeRabbit and Claude, not two Claudes

CodeRabbit and Claude code-review each catch different things. CodeRabbit is strong on style, conventions, CLAUDE.md adherence, and security smells. Claude review traces control flow and finds deep edge cases beyond the diff. The deeper reason for using two different systems is that they share fewer blind spots than two instances of the same model. Uncorrelated errors are the whole point.

You could replicate most of this with two Claudes given very different prompts, one as a style auditor and one as a logic hunter. CodeRabbit just comes pre-built at around $15 a month per repo. It is replaceable, not magic.

Who fixes the comments

RoboBun, the author bot, fixes them. The reviewers comment. RoboBun reads the new comments on its own PR, pushes fix commits that address each one, and resolves the threads.

RoboBun opens PR
  -> CodeRabbit:    "this is O(n^2), use a map"
  -> Claude review: "misses the null case at line 47"
  -> RoboBun reads both, pushes a fix commit, marks both resolved
  -> repeat until reviewers stop commenting

Boris’s line is that replying is performative and fixing is the work. The bot does not post “good catch, will fix.” It just pushes the fix. Reply with code, not with words.

5. CLAUDE.md as operational memory

CLAUDE.md belongs at the project level, not the global one. The global CLAUDE.md captures personal preferences. The project CLAUDE.md captures project-specific facts that, if missed, cost a PR cycle.

A project CLAUDE.md should include how to build the project, how to run the exact changed tests, how to run full CI locally if possible, where tests belong, how to name tests, common mistakes agents make, an architecture and folder map, commands that avoid stale builds, the expected PR format, and the definition of “done.”

A concrete example from the talk: the Bun team noticed Claude kept writing tests where the assertion message hid the actual error. They added a paragraph telling the test harness to print the error before the less informative assertion fires. One line of CLAUDE.md, and every future test the agent writes is debuggable on first failure. Boris calls this compound engineering.

6. Self-verification

The headline claim is that agents can only run autonomously if they can verify themselves. Before saying “done,” the agent has to be able to prove the change works. That means it needs the abilities to:

1. Run the build and see errors
2. Run tests and see pass or fail
3. Run the actual app, hit the endpoint, see real output
4. Read CI (gh run view <id> --log-failed)
5. Read its own logs while testing

For UI work, add:

6. Take screenshots (chrome-cdp or playwright)
7. Compare before and after visually

Bun gets this almost for free because as a CLI the binary itself is the runtime. You compile it, you run it, the behavior is real. The practical move for any project is a verify script that runs the chain end to end, referenced in CLAUDE.md as “before saying done, run ./verify.”

7. Hill climbing

This is a prompt pattern, not a feature. You give the model a target metric, a way to measure it, and permission to iterate. Then you walk away in auto mode and come back later.

Goal:       make X faster than Y
Measure:    run `bench/sharp-comparison.ts`, parse the output
Budget:     keep going until 20% faster than baseline, or 50 attempts

The loop is straightforward. Run the benchmark. Hypothesize an optimization. Implement it. Run the benchmark again. Keep it if better, revert it if worse. Repeat.

In the demo Jared said “make this image library faster than Sharp,” gave a benchmark on a Linux box and a couple of hints, and Claude iterated to the win. Boris says 4.7 is the first model efficient enough to do this day to day, and the pattern is badly underused. The /loop skill can drive it.

8. Tooling worth flagging

Auto mode auto-approves permissions. It is the real fix for “I left for an hour and came back to Claude waiting on a yes/no prompt.” Replaces the riskier --dangerously-skip-permissions and makes overnight sessions possible.

No-flicker mode (CLAUDE_NO_FLICKER=1) is a renderer rewrite with virtualized scroll and selection. Constant memory and CPU, mouse clicks work in the composer. Launched on April 1 and looked like a joke. Boris thinks it should be the default.

/loop for monitoring. Jared had Claude watch a PR on a 20-minute wake interval and admits that is too long.

9. Generalizing beyond Bun

Most companies are not open source, so “GitHub issue to bot” does not apply directly. The translation is:

customer support ticket -> Claude bot -> reproduce -> PR -> adversarial review loop

Same shape, different inbox. CLI and systems projects are easier to automate because reproducibility is strong (command in, output out, architecture-specific tests). UI apps need equivalent verification with screenshots, video, or browser tests.

10. Action plan

A. Adversarial PR review. Extend /land to spawn two reviewer agents (a style auditor and a logic hunter). The author agent fixes both rounds before you see the PR. Or install CodeRabbit on the repos.

B. The “fails on main, passes on branch” gate.

git stash; git checkout main
bun test path/to/new-test.ts   # must FAIL
git checkout fix-branch; git stash pop
bun test path/to/new-test.ts   # must PASS

Belongs in the reproduce-bug skill as a verification step before declaring success.

C. CLAUDE.md as repeat-detector. After each session, ask: did the agent make a mistake I corrected? If yes, one line into the project CLAUDE.md. Run claude-md-improver monthly per project.

D. Hill-climb skill. A small /hill-climb wrapper. Inputs: goal, measurement command, threshold. Then it auto-loops. About thirty lines.

E. RoboBun-equivalent for Linear. Watch a Linear label, sandbox, repro, failing test, fix, PR. Biggest payoff, most upfront work.

11. Codex timeline and key ideas

Timeline

• 1:27–2:34 Setup. Bun uses Claude Code to build and maintain Bun. Jared starts agents live to fix GitHub issues.
• 2:05–3:10 RoboBun reproduces every new issue. Can only submit a PR if it includes tests.
• 3:11–4:09 Bottleneck changes. Not “can we fix it?” but “is this the right fix and should we merge?”
• 4:12–5:45 CodeRabbit and Claude review loop with RoboBun. CodeRabbit catches style and convention. Claude catches deeper edge cases.
• 5:48–6:17 Agents remove review friction. Fix lint, push, resolve comments without human context switching.
• 6:19–7:51 Bun suits this because it is CLI and systems code, reproducible. UI products use screenshots or video for verification.
• 7:16–7:51 For private companies, replace “GitHub issue” with “customer support ticket.”
• 8:11–10:00 CLAUDE.md is critical. Every repeated instruction, build command, test convention, and prior mistake goes in.
• 10:20–10:59 Agents need access to CI errors, build logs, tests, and code, to complete the loop before a human reviews.
• 11:01–12:53 The vision is hundreds of agents in parallel. It only works if they self-verify.
• 12:56–15:39 Live PR inspection. Jared checks style, waits for Claude review, explains when he trusts tests.
• 16:29–17:39 The new bottleneck is confidence. How do you prove the PR is correct enough to merge?
• 18:10–20:21 Larger work is possible. HTTP/3, HTTP/2, image processing, benchmarks. Give the model goals, let it iterate.
• 20:23–21:39 Hill climbing. Metric, measurement, permission to iterate until improved.
• 22:15–23:55 Tooling. CLI, auto mode, no-flicker, virtualized rendering, mouse support.
• 24:00–24:47 During the talk, agents produce multiple PRs live.
• 25:03–26:13 Bottlenecks keep moving. Code, tests, deeper verification, planning.
• 26:14–27:31 RoboBun can sometimes handle feature requests. Jared is careful because features need taste.
• 27:33–28:43 Agent PRs become suggestions. You can reject them without social cost, which raises the merge bar.
• 29:13–30:08 Auto mode lets long-running agents work for hours without permission stops.
• 30:20–30:54 Closing. Teams are still figuring this out by constantly finding and automating the next bottleneck.

Codex’s most important ideas

1. RoboBun is not just “AI writes code.” It is an automated engineering loop covering read, reproduce, test, PR, review, and CI.
2. Tests are the admission ticket. A PR must include tests that fail on the old version and pass on the branch.
3. The real bottleneck becomes merge confidence. What proof do I need before merging?
4. Agent review should be adversarial. One catches conventions, another traces code paths.
5. CLAUDE.md is operational memory. Correct a mistake once, write it down forever.
6. The agent needs the whole feedback loop. Without access to CI, tests, build, and branch, it dumps half-finished work.
7. CLI and systems projects are easier because of strong reproducibility. UI needs screenshots, video, browser tests.
8. Hill climbing is powerful. Give a measurable target and tools to iterate.
9. Auto mode changes the workflow. Sessions run overnight without permission stops.
10. Planning and taste are still human-heavy. Whether Bun should have an image API is a product call.

---

Analysis of the “Bun × Claude Code” live coding session. Claude’s explanation combined with Codex’s timeline and key ideas.