How Claude Fixes It's Own Code

We've spent a ton of time recently talking about how to build as effectively with agents as possible. But I don't think we've talked enough about the communication around it. How do we work together as a team? How do we review code? How do we choose what merges and what doesn't? How do we manage all of this? Especially at the scale we can ship at now with tools like Opus 5.5. As you can imagine, anthropic teams like the Cloud Code team are experiencing this at a scale that is hard to comprehend. They can't even use Git and GitHub anymore because they slow them down too much. So, how does a team possibly review all of the change requests from other people? How do you manage a poll request load like this? We've been experiencing this ourselves with T3 Code as we've been getting as many as 800 PRs filed a day. Figuring out how to get through all of that with a small threeperson team is nearly impossible. This is just a different world we're in. And the idea of how code is reviewed and landed is and should be done in a fundamentally different way. And that's what we are here to talk about today. How do you handle this level of load, this burden of constant requests for changes being made on all of our repos now that Opus has made it as easy as sending a prompt to get a pull request? How do you decide? How do you manage? How do you deal with all of this? I have a feeling this is going to be a fun one. I've actually not read this yet because I wanted to read it with y'all together. And I'm hoping we can learn some things too here because this isn't easy and figuring this out is a a really important piece of getting

this new era of engineering right. I also think I should be able to share some special insights at the end on how we were able to ramp up so much with T3 code. I'm very excited to show off more of our process cuz it's been getting really fun. Julius is cooking like I've never seen. But I do have one other cool thing I want to show you first. Today's sponsor. When you think about it, it's kind of crazy how much more powerful agents have gotten recently. And it's not just cuz the models have gotten smarter. The majority of the wins have come from the fact that the tools have gotten so much better. And more importantly, the models have gotten better at using those tools. Now that agents can do so much more on our behalf, it's important that we give them what they need to do those things. And when we see tools like Grockbot, Muse, and more all taking over our day-to-day lives, they need to be able to do things on our behalf. A lot of those things are on a browser. 85% of the APIs on the web are not accessible to just be hit over curl. You need to actually use the website in order to do things. You could build that all yourself or you could use today's sponsor, Browserbase. These guys have built the best portal for the web for your agents. The simplest way to think about it is that they turn everything on the web into a programmable API. Your agents need search, they have you covered. Do you need to fetch content from a page in a format your agents can use? They have you covered. Whether it's HTML, JSON, or markdown, they can get it from the web.

And of course, if your agents want to do things on the web, like click buttons, navigate, fill out forms, clear captions, and more, that's where the browser as a service part comes in. There is so much work blocked by the browser. And if you give your agents the ability to do it, you'll be amazed at what they can do. If you don't have any ideas yet, just go ask Claude. Say, "Hey, look at what I do every day. See how much of this can be automated." I bet you'll be surprised. And if you are, start today at soy.link/browserbase. Now that's out of the way, we can dive in. Let's start with how the cloud code team does code review and then see how it compares contrasts to the way we do it and what my team can learn from it because I'm actually very excited to possibly improve our process here. Starts with the obvious question. How do you do code review at the volume of code that agents are producing on the cloud code team? We once had high coverage in deep reviews. We started optimizing for the minimum amount of human review time that doesn't raise reverts or incidents. Sounds about right. Although this sentence is horribly formatted. Let's see how this goes. I've said before that Claude writes around 90% of Claude Code's own code. It's got to be more than that at this point. I don't think I or anyone on my team has manually edited a line of code in T3 code other than like pros in markdown files in at least 3 months, probably longer. Just Claude is probably over 90% of our code and the rest is codecs. Yeah, I I am confused by this. I I would have thought it'd be more by now. Previously, the numbers were insane. Like we barely touch the code by hand. It's 85% of our code is

claws. Like that's insane. A year later it's 90. That feels low. Anyways, how do they review code at the volume that they're producing it? Does anyone read the code anymore? And if something breaks, who catches it? This post is meant to cover the code review flow on the Cloud Code team and how it has evolved over time. From June to September, merged PRs to Cloud Code's codebase have more than doubled while the share that had to be reverted fell by more than 40%. This feels as much like model improvements as it does process improvements. Like our process improvements have given us some bigger wins than this, but model improvements have given us even bigger than that. Like I would say our revert rate has dropped much more than 40% since Opus 55 dropped. Personally, we also had another big drop when I banned all Gemini models from landing code. Addy and crew point out that they didn't remove. They did move most of the reading of the code to machines. optimized for human time and attention and invested heavily in making it fast and easy to undo mistakes. Is this going to be a safety nets not guardrails? Please tell me it's a safety net guard rails. That's going to make me so happy. I have a good feeling about this one. So, we start by rethinking the purpose of code review. When agents write code, the number of changes go up fast. In Anthropics Monoro, changes landing per day went from 2500 a day in May to more than 30,000 in September.

Cloud code lives in its own repo separate from the monor repo, but it's seeing a lot of pressure, too. When you're shipping this fast, manual line by line code review is incredibly difficult, if not impossible. It's also interesting to know that cloud code lives in its own repo. That means they were considering open sourcing it is what that means. So, code review has two main jobs. Find and fix issues before they ship and more importantly, in my opinion, building a shared context and codebase understanding across the team. This is the most underrated aspect of code review and I'm still working on figuring out how to maintain this as the pace we ship increases. Like how do I make sure my team understands the crazy changes I am making to our wire protocol? How do I make sure that Julius's changes to our potential integration of computer use are understood by the rest of the contributors? How do I make sure everyone gets Orchestrator V2 before we merge it? We had random contributors who were pushing Orchestrator V2's like branch really hard before we had landed it that understood the overhaul of our orchestration layer better than I do today before it even merged. That is a thing that's going to be hard to balance over time and I'm excited to see where it goes. So, let's see what their recommendations are here. Point of this article is mainly to focus on finding and fixing issues, but we'll come back to the second part near the end. Okay, I can wait. They personally believe that quote did a human read the code end quote is a very imprecise proxy for quote what could go wrong with this change in which check mdash automated or human mdash is the right one to catch it. This is an important detail. I I

feel like the conversation around code review has eroded horribly because it was based around this idea that agents make mistakes so much more aggressively and actively than humans do. So, if you don't have a human checking to make sure the AI didn't make some dumb obvious mistake that the code is safe, otherwise you're going to just ship slop. I'm going to be so real with you. Pretty much every single regression we have shipped in either T3 chat or T3 code, whether or not I reviewed the code, I would not have caught it in code review. And I promise you, not a single one of you would have either. The types of bugs that get through now are not the ones where just staring at the code for a long time, you're going to be able to deduct what could go wrong. You need lots of other ways to analyze and verify your code. I also have the hut take that most agents are now better at code review than most humans. And if they're not better in your codebase, that's cuz your codebase sucks, not cuz the agents suck. But that's a conversation for another time. Asking that question also changed when reviews happen. Before, review was a gate at the end of the road. You write the code, open the PR, and then you wait for your colleagues.

Now, reviews are running constantly, more like a spell check while you write. Most changes have been reviewed and fixed by Claude at least once before a human is asked to look. That shouldn't be most, that should be all. I was saying this for a bit. If you're putting up PRs that your agent hasn't even reviewed yet, you are wasting the time of your team, and it upsets me. Apparently Claude also raised their remaining 10% but they have to fix up their attribution to be super confident before they claim that. Good luck. That's hilarious. Here's a fun example from chat in my chat of the like bad code makes it through review. There was a bug in LVM that broke safe Rust for 10 years. Some of the most well-reed code by some of the smartest engineers in the world left an aggressive bug in simply because the best humans still can't catch everything. Humans and agents are both lossy. The main difference here between the smartest AI and the smartest human is that the smartest AI isn't going to get burnt out if you tell it to check over and over and over and over again. The humans that are smart enough to find these things are also busy enough they're not going to bother. Oh boy, we're about to get into my favorite part of this conversation. The only truly scarce resource right now other than GPUs, human time and attention.

We're going to get into all my favorite themes I get to talk about in my other videos here, aren't we? This is going to be fun. We want to preserve human time and attention for issues that only humans can solve and make sure automated checks are catching the rest. At our current rate of change, there is still an incredible amount of work for humans to do and our team encounters new and novel challenges almost daily. I agree here fully. I would add one other thing. Human energy is different from time and attention because certain things demand more energy and less time than others do. and figuring out how to minimize the energy cost to keep humans from doing things that feel painful and slow and miserable so that the humans are spending time doing engaging and important things is a valuable distinction. I'll go more into detail on that in a bit. Human time covers both sides of a review's cost. The time people spend reviewing and the time people spend fixing what bad code breaks. There's another aspect of time here which is the time between all of these parts which is human files PR.

Time is waited for. Another human comes in and reviews it. Time is waited for. The original human sees the review, reads it, goes and makes changes, pushes out the PR. Time is waited for before another review happens. That is a lot of time and more of it's being spent waiting and the in between of coms than is being spent actually doing the work. And this is one of the biggest bottleneck that prevents things from shipping. So even if you can make it so less time is being wasted doing the code review. If time is still being wasted fixing the code, it doesn't really matter that much. So the goal is that most pull requests in the cloud code repos for the CLI, the desktop web apps, and the mobile apps should be able to merge with at most one human in the loop who can be the author with no increase in reverts or in incidents. To measure human time, we track human approvals per PR, human comments, and how long a PR waits on a person. Revert and incident rates are the guardrail. If they rise, we haven't saved human time. We've only moved it. So, let's look at the life of a change. Typical change from an engineer's laptop or a cloud session in Slack goes through multiple layers of code review. So, after the change is initially made, the work is verified.

This could be Claude running tests, building and linting, maybe has a second agent come in and review the diff in a fresh context. I would add where possible the work should be verified beyond just like a quick test check, build check, lint check. If the change is complex enough, it is almost always going to be worth it to spin something up and really test it, especially for like UI work, whether you're spinning it up in a simulated browser or you're spinning it up with computer use in an environment where it can open up the app and actually check things. One of the things I've been loving with T3 OS is that I made it easy for each thread to spin up its own like window in like the equivalent of in Mac OS how you have the different workspaces here. I made it so every thread can get one of those. It has made it very easy for a thread to spin up an Electron dev server for T3 Code, record a video of the change, showing it working, and then send it to me or put it in the PR. I also added the ability for during dev to have fake model selections that have pre-baked events that they will send through the pipes so that an agent can run a full end toend test without ever having to actually burn tokens beyond the tokens it's burning doing the like visualization of the check. So it can test our code without burning tokens.

And all of these types of things have actually helped a lot with our confident shipping and the frequency that I can land scary changes because my agents now can actually test and validate the code the same way a user would. So by the time my agent is telling me the code is ready to be looked at, it's already verified it in most of the ways I would have verified it. But that's my process, not Claude team's process. Let's hear more about theirs. On every poll request, they go through the required CI which includes lint types, tests across platforms, building a packaging, red blocks, the merge expected. Then they have the cloud code review step where cloud code itself comes in and runs through this like a human would play with it. It spins up multiple agents in parallel hunting for bugs, verifying changes, using verifiers to vote on each finding. Survivors are ranked by severity. Then there's security review, which is a separate pass where they find things that could be potentially insecure. And then there's a risk based approval. They have a classifier for how risky is this change and how much human intervention should be used here. No person, one person or two people. This is interesting. I've been thinking about doing something similar in T3 code where in like the template or the like skill we have for filing PRs, I'm considering having it put a blurb at the end of every PR saying on a scale of 1 to 10, how impactful is this change? How many users are impacted by this change? And how risky is this change? to make it easier to digest the PRs when they go up. Then we have a merge queue. Uh we don't for T3 code yet. We probably should. We've broken main enough times.

Not broken in the sense that like the build is broken, but broken in the sense that like one or two random CI things fail. We probably need a merge queue at this point, but we don't have one, and I would like to avoid it as long as I can. And then after the merge, the changes will be behind a flag that's used internally initially. Then they'll do a gradual release. Then they watch this release to make sure everything goes well. fix forward if it's broken or roll back if it's too broken to just fix. That all makes sense to me. I like this framing of layers where you want the good change to make it through and you want all of these blockers to block things that aren't going down the right path. So maybe a change that is bad leaks through the verification system but then fails the CI layer. Maybe it gets through CI and then fails the AI review. Maybe it gets through that and then fails the AI security review. These layers increase the likelihood that bad code is filtered out before it makes it to the bottom. But there's another issue here that is worth noting. Each of these layers isn't just trimming out bad. It's sometimes appending additional unnecessary stuff, too. I cannot tell you how many times I've seen a PR that should have been like 50 lines of code get bloated to be 800 to 2,000 because the agent in the center addressed every single piece of feedback from every single layer on the way. And the result was just bad. And the most important thing is at the bottom here. If something bad gets through to the end, you need to be able to roll it back fast. This is like the most important thing. Failures should affect as few people as possible and be as cheap as possible to undo. This has always been the case, but it is worse now than ever.

Everybody's systems needs to be set up so roll backs are viable. So step one is the work is verified before anyone else looks. Most reviews are happening before the PRs even exist. We give Claude something it can run like a test build, screenshot to compare against the design and ask it to show the output instead of saying it's done. Then a sub agent reviews the diff with fresh context. Both habits come from their best practices guide. Before opening a PR, many of us have also run a few internal skills. One hunts for bugs, one simplifies the diff, one checks the behavior end to end. UI changes also get checked against our design guidelines. Yep, this is aligned with what I was talking about before. Boris who was the original creator of cloud code has given claude ways to verify its work and says that this is the most important thing for getting good results. This is I absolutely agree. I think a lot of the people who are struggling to get good results out of working with agents are struggling because they haven't given the agent any way to check if what it did worked or not. And to be fair, a lot of their co-workers, especially if they hired a new edge, they'd probably run into a lot of those same problems because if you make code that is hard to check, then you made a codebase that sucks to work in. Boris estimates that this makes the final result two to three times better. And he writes the following. Verification looks different for each domain. It might be as simple as running a bash command or running a test suite or testing the app in a browser or a phone simulator. It could also be significantly more complex. Like I've had to build some crazy VM for testing T3os. Tests are not easy always.

Building these systems to verify things are not easy always. But you got to figure out a way to do it. There's always some way. And if you notice the automated reviewers are consistently flagging the same things, then it's time to fix at the source. You can fix this in a skill with a lint rule or with updates to instructional files as such as a claude MD change to tell claude what the project is and like what conventions to follow. A review MD which is written by the code owners that tells the reviewer what matters in their part of the code. We check separately whether the reviewer actually follows it. The good ones are short. They say what failure looks like in that repo. For example, in the docs it could say it lies. For config it says it misroutes. You can cap the nits list paths to skip yada yada yada. find good place to put this info. I will say this is not the right starting point. I'm gonna cite some of Lauren's stuff here because I think her guidance here is a little bit better. When you have to correct your agent, you should look for the place closest to the top to fix what caused the agent to do things wrong. The top spot is the code base. If it's making mistakes in the code, see if you can fix the code to make it less likely to make mistakes. Don't do this by putting comments everywhere. Do this by making the architecture better and easier to work in. Then static analysis is a great next step. Things like llinters, checkers, rules, type checks, all those types of things that can give agents or even humans feedback that the thing isn't implemented correctly. That gives the agent the feedback it needs to do it correctly if it couldn't figure out what correct is from the code itself. Then if both of these fail to catch the problems

and you've tried and been unable to make it so these specific problems cannot be fixed at those two layers, then you bring in pros. That's when you start touching claude MDs, review MDs, agent MDs, you start building skills, you start doing these things. You do that after the rest failed. But you should always be trying your damnedest to make it so the code base and so the lints, checks, etc. guide humans and agents to success. The silliest way to frame this is imagine a really, really smart human that just joined your team but doesn't know where anything is. You should make it as easy as possible for that person who has worked in thousands of code bases in the past but has never worked in Europe specifically. Should make it as easy as possible for them to just blindly dive in and work and minimize the chance they make dumb mistakes. And just imagine this person doesn't care about TDM and will do tedious work all day every day without reason. If you were already setting up your code bases for success for your teammates, this transition should be very easy for you. The next section that Addi has here is AI review on every PR. Each poll request gets a claw review before a person looks. It reads the diff against the whole repo and the PR's stated intent and it asks, "Does this change solve the problem, fit the system, or does it break something else?" On a bug fix, it looks for the root cause. The public version code review for cloud code is now generally available on the team and enterprise plans and/code review runs a lighter version locally. When we launched the research preview, the share of internal PRs getting substantive review comments went from 16% to 54%.

Most PRs over a thousand lines get findings, and most small ones come back clean. Admins set each repo to quick, standard, or deep. Deep is worth it when one missed bug costs more than the extra review time does. I'm not going to pitch the cloud review stuff cuz I think their review is way, way, way too expensive for most real world cases, especially when you're shipping as fast as they're recommending here. There's lots of other review tools that are cheaper or can be used subsidized locally. One of the things I love is my council skill where I have my code get reviewed by three different models from three different families. That gets subsidized. I'm using my subscriptions for it and gets as good if not better stuff than these code review processes do. And it runs on a real computer so it can spin up the code and check. So I don't think you should go use their product for it. It's not bad but it's expensive for what it is and you could just use better prompts and other agents. And I think the multimodel side here is an aspect they'll never talk about that I do think is really good. Another important piece, and I'm somehow surprised people haven't caught on to this part yet, but I know a lot of you haven't, and it's really important. We shouldn't just have agents and review bots leaving a bunch of PR comments that we go through by hand and choose which ones to act on and which ones to not act on. That's the job of your agent. Comment to my sponsors cuz I know y'all watch my videos, especially ones like this, and I know a handful of you guys are doing AI code review products. Get the goddamn prompt suggestions out of the code review. That is 2024 core. We're in 2026. I'm not

copy pasting from the code review back to my agent. My agent's babysitting the PR. My agent is deciding if your feedback is good or not. All of the code reviewers do this. Even like Bugbot, which you would think from cursor would understand this. Nope. All them by default leave the one-click AI fix prompt suggestion that you copy paste over. No, that. And and not the autofix stuff either. Get rid of all that. I want my agents to decide if your feedback is good. And no human should have to look until that is done. I should not be seeing the comments that my agents leave on the PRs. So, please make sure that the findings are verified by other agents before you force humans to review them. Several agents can look for bugs at the same time and then a separate group votes on each finding and only posts them if they're enough agreements that this finding is real. Findings that are only about style, diagnostics, or cleanup get dropped. We AB tested these filters on our own PRs before we kept them. What's left shows up as inline comments sorted by severity. Each is tagged as important, nit, or pre-existing. So, a bug that PR just didn't introduce doesn't look like the author's fault. Every finding that gets through this filter does have to be answered. You either fix it or resolve the thread with a reason. Often, an agent picks the fix up overnight. So, the author comes back to a PR that's already been through another round. Is Anthropic moving slower than us? I'll have a PR go through like eight plus of these rounds in an hour or two and then bother me when it's ready to go. Next section is security reviews that can block. Security gets its own automated review too. A PR can't merge while any

finding from the bug review or the security review is still open. You either fix it or you close the thread with a short reason which is how you clear the false positives. If a later commit fixes the issue, the findings clear on their own. You can set up the same thing with open source security review, GitHub actions, or the /security review command. Setup is covered in their guide. Cool. Awesome. Now, we have automated checks that block. This is the usual CI type checks, list, linting, testing, all those types of things. You guys hopefully know what those are now. And we have the human approval layers where low risk means no human is required. You can just merge this when it's green. Medium, which is an ordinary change. You should have one person look. high is this is severe so at least two people should look e I don't love the 012 way of framing this I would rather we and our agents can decide on a more dynamic scale I am going to play with these ideas though so we will see going forward but the point of the tiering is that it doesn't pick an author it picks the risk level a lot of the bots I've seen are trying to choose which human to tag in and be like this person has to review this thing this person has to review this thing Ideally your team trust trust each other enough to know who can review code. And if one person comes in and reviews it, they're like, I should have this other person look instead, they can. But the bot shouldn't be the one tagging specific people. The bot should be describing the severity of risk and then the humans can do the rest. Friends from cursor are saying that risk hearing is a losing battle.

They're still doing it, but they have not had as good of a time with it so far. They call it that there are some paths that always need a person. Anything owners have listed as sensitive in any CI workflow files. Yeah, checks out. I have a couple of those in T3 code. Like anything touching scroll areas, that's mine. That's not merging until I look. Next section is that only trusted authors are getting the lighter paths. Only authors of the repo's trusted list get the light paths, mostly the team that owns it. Everyone else needs a second person to approve. An agent's approval never counts as a person's either. Automated review can clear a low-risk change, but it can't fill a slot that calls for a human. This is one of the things that's been pissing me off on GitHub a lot. I want my bots to be tagged differently than me. I've done this in the sense that I added skills where they put a big call out saying I am opus 55 via cloud code and T3 code leaving this comment on Theo's behalf. But I want to see when Theo's bot approves separate from when Theo approves. I want two different approval checks for that. They also say that whoever is prompting for these changes is the person on the hook. You can't blame the agent if you prompted it.

That's your ownership. That's your problem. And approvals only cover specific commits. If new commits land that the system can't vouch for, like an agent push or merge someone resolved by hand, the approval has to be restarted. They have a merge cue with flags and gradual releases. They start with the latest channel first, then stable gets the changes after. Fast detection, fast roll back. Some mistakes will still get through, so the last layer is about catching them fast. They have a weekly on call rotation. Incident review asks which layers should have caught the issue. Noticing a bad merge still depends on the author and on call as well as the revert and incident dashboards. We don't yet have circuit breakers that tighten approval when those numbers move. Engineers here want them before more code moves to zero human merges. That makes sense. So, they're not doing much zero human. We're trying to do a bit of this now for easy enough bug fixes. That's how Julius is able to start landing like 400 PRs a day. One colleague said that slowing down lowers your error rate, but mostly just cuz you're doing less of everything. Teams decided that they'd rather get faster at noticing and fixing problems rather than slow down to prevent them. And I agree there. If your solution to not shipping breaking changes is to just go slower, then you're not actually solving the problem.

Banger. So now a more important section. How do we actually improve the quality of the reviews and the reviewer? The code review process is constantly improving in a loop. Any incidents become eval. The team collects past PRs that cause incidents with their fixed forwards and reverts and had engineers who know the code confirm each bug and where it lived. The reviewer runs against that set daily. Not how I would do that, but I like where they're going here. So, there's an incident, an expert checks and confirms, they add it to the set, and then they have this daily eval run where they check to see if these things would have happened again. Cool. We measure noise by watching what people do with the comments. It's easy to catch more bugs if you don't mind flagging a lot of non-bugs, so we need to check on that. Absolutely agree. Noise in PRs as hell. They sort every comment that the reviewer leaves by how the person on the PR responded, if they fixed it, or if they said that was wrong, or they said the behavior was intentional, or if they decided it didn't matter. We also run the reviewer on PRs that they know are clean, and they count how often they report something that would have led someone to make a harmful change. Our priority, written down, is catching real bugs first and cutting reviewer rounds second. That makes sense. Noise costs more when the author is an agent. A person can wave off a knit in seconds.

an agent will fix it, which means another commit, another round of review, more unnecessary code, and so much other slop. But so the reviews do have to converge. An agent author and an agent reviewer can keep each other busy indefinitely. I had what I thought was going to be a simple change. I'm working on the like Grockbot style experience in T3 Code where there's like one main thread that controls all the other threads. I had a working prototype that wasn't quite how I wanted it to work. So I threw it out into council review and said, "Here's what I want. keep iterating until the whole council passes. And then I forgot about it because it was folded in my working tab. I just noticed like yesterday it's been going for four days. The PR that it's working on has gotten to like 80,000 lines of code and I had to hop in and be like, "No, you're just running in circles right now." And I'm realizing I need to add short circuits to some of my skills cuz it's just it's not making any progress at this point. So yeah, you need the reviews to converge. You need the things to to end eventually. One CLI PR went through more than 50 rounds of review and fixes overnight before the author stopped it.

I didn't even read and say that when I was describing mine. That's hilarious. Yeah. Anyways, when we ran get blame on what the reviewers kept flagging, about 60% of it was code that had been there since the reviews first pass. The reviewer was rarely finding new bugs. It was finding a few more that the old rounds just didn't catch. We made it worse ourselves. We' gotten used to running several rounds by default since each pass was catching something new. This can also be worthwhile on risky changes, but not for every PR. The team now tracks rounds to pass alongside recall and changed re-review. I did also manage to fix this in a silly way. Instead of just tracking all of the nits and going until the agent said, "No, this is fine." I moved over to a scoring system where the review agents leave a score 1 to 10, and I pick a bar for what score is high enough. Because a PR might have 18 comments but still be a 9 out of 10 cuz all the comments are just silly small knits. But you might have a PR that only has one comment that's a 6 out of 10 and the one with one comment is higher priority to be fixed than the one with 20 comments in the state. So finding a way to to set a bar that isn't by how many comments are left but is instead by how much confidence can you build in the system. That is the path that works. So they started with a round cap. I also did this with soloop. I haven't done this with council yet and I need to. After three rounds, the reviewer will ask the author to step back and reconsider the approach.

There's a diff only re-review after the first few passes. There's a layer to flag the class of bug so the author can fix every instance, not just one line. And then no repeats of findings that have already been posted. These are good changes. Review agents also are running nightly over existing code, helping catch old bugs before they show up somewhere else. After making the convergence changes, they saw a 13% increase in PRs that were passing after one review and fix round. They had a decrease for PRs that need five or more from 9% to 1.5. And the mean reviews to pass went from 2.08 to 1.32. These are good changes. So, where do humans stay in the loop with all this? The thing that we're all curious about, where do we go? Actually, one more good piece of feedback from Jake over at our friends at Graphite aka Cursor aka SpaceXi aka Grockbot. He thinks that scoring is not the solution. Agent should be able to be trusted enough to make a boolean decision with the right context just like human engineers can who you pay to do that. Quote, can I ignore this comment or not? This is a good solution.

His goal here is to keep it simple. I largely agree. I do need to experiment with other options here more. I kind of just if I'm not happy with one, I play with it until I have an improved solution and then stick with that until it pisses me off. I need to be doing more of the like different tries and ways to do this. So, uh yeah, shout out to all my friends at all of the labs who come in and give feedback as I record these. It helps a ton with getting this all right. So, where do the humans actually come in here? They decide whether the thing is worth building or not. Direction, taste, and scope. One team that declared its review bottleneck solved still kept human reviews for changes from outside the team. So someone asks quote should we even do this? Next we have reviewing risky code like billing autoupdater and CI workflows. Reviewing architecture because this sets patterns that others copy. Reading the evidence test output commands that run and what they actually return. Screenshots review findings things like that. If you can't say what the checks would miss, ask for a real review rather than a quick approval. And of course owning the outcome. Whoever put up the PR is the person who owns it.

It doesn't matter where the code itself came from. So, how do we share all of this context across teams? Because again, this is the thing I love code review for, not just catching bugs. I love code review for making sure everybody knows what's going on in the codebase. And to this day, like when I ask a teammate to review code, it's usually for this reason. I want them to understand it because if they don't understand it, it means I have to fix things. And if they do understand it, it means I'm not solely responsible going forward. Code review second job's always been to spread codebase understanding. Reading a teammate's diff is how you learn what changed and how the pieces fit. If machines do most of the reading, how does anyone learn the code? We don't have a formal process for this. Nobody on the team keeps a complete mental model of cloud code in their head. I know you, some of y'all are going to think this is insane. Like, someone has to know how it all works. I hope unemployment is treating you well if you feel that way because any sizable codebase does not have one person who knows everything. The code bases that are, and I've got in arguments with this, and I I've went back and forth on it. There are lots of really important code bases that are well understood by the one or two people that maintain them that are very small. Those aren't full-time jobs anymore. If your codebase is small enough that one person can understand it, then that codebase doesn't have enough throughput, not in terms of traffic. Like there can be a small codebase that does billions of requests a day, but it's not changing enough that it is that important in the sense that like you need to build a super religious process around it so that you can make changes every day.

Like if you're only making one change a week, spending five of the six days you're working making sure the change is good. That's fine. That makes a lot of sense. It's just not the types of work that we're talking about here at all. So, as I'm saying, most full-time jobs are working in code bases that no human can understand all of and no agent can understand all of either. Nobody needs to understand it all, though. When someone wants to understand a part of the codebase, they can ask Claude to write an explainer for exactly that slice. It takes minutes and it reflects the code in its current state. I think people severely underrate how powerful this is. When I'm reviewing heavy changes, I'll often just paste the PR into T3 code with Claude and ask it to give me a breakdown of the changes. What is changing? Why is it changing? And where is the risk? Some people on the team go a step further and have Claude build an artifact to visualize the part of the code to teach them how it works in whatever way that person likes to learn. This is awesome. I highly recommend doing it. If the goal is to spend human attention well, then this hugely beats out reading a trivial diff line by line. Oneline copy changes teach you nothing. Explainers for the module you're about to change teach you a lot.

Our culture of defaulting to working on open Slack channels is helping here too since you see what others are building while they build it. I do love that. It's one of the things I really hope we can figure out in T3 code is how to like work together as a team and see who's doing what and when. There's a lot of reasons it's hard. I want to figure that out though. And then of course the you land it, you own it. If you land a small or medium change or and something still goes wrong, it's the team norm that you're going to be the one to fix it. A few things keep this from going badly. Changes are small so they're easy to review, debug, and revert. Everything's reversible. Blast radius sets the bar. Expensive code gets more people in more checks. Regressions surface fast through telemetry and internal use. And it isn't blame. The ownership doesn't mean you're at fault. It means you have the best understanding. So you are ready to come in and fix the thing you have the most context on. Ownership is also shared. Much of the team works in open channels where anyone can see what the others are building, pick up threads, or point a claw at a gap. And feedback comes from people who don't own the code as well. That openness is a big part of how quality stays high without gating every single line. Here's another big piece. I love they put this in here. Changing direction. Agents make it cheap to start things. So, some work shouldn't land. I would guess that even if my PR is a T3 code, about half of them merge, the other half get closed for various different reasons. Usually someone in the channel asks why, points at an earlier decision, or suggests a simpler route, and the author closes or reshapes the PR. Since the code is fast to write, closing the PR doesn't feel like a loss.

The trick is to raise a different direction in the thread early, not as a blocking comment or a finished PR. So, how can we adopt the processes that Anthropic is sharing here? God, I hate how I don't know who wrote this. Like, I know Addy wrote a lot of it, but some of the sentence formatting here is awful. This is one of the worst sentences I've ever read. Most of the pieces to do this with your team exist today. There is not there's no pause or sectioning of this sentence at all. This needs a comma or two or to be split into multi multiple sentences so badly. Sorry, I I'm picky about pros. I I like writing. Crazy how large language models have gotten good at pretty much everything other than generating text. I hate LLM writing so much and I am I'm smelling things I don't like here. But we can still learn from how to adopt the process. I like that we start from before the PR because there is before, during, and after the PR, like the sections you should think about this stuff through. Before the PR, you should give your agent checks it can run and ways it can verify its work. You should run the code review and security review skills locally. I just have this as part of my like main skill for landing changes and the agent decides what to pull in and not pull in. You add a review MD and more repeat review comments can go into that. You should fix the code first. On every PR, you should require CI, add testing, risk, how to revert to PR templates. I like this part. Like your PR template and the skills or whatever you use to open PRs should include how to test, what the risk is, and how to undo this change.

Add an AI reviewer. You can use one of theirs that they suggest or any of the awesome sponsors of the channel on the repo. Tune the review MD for a few weeks and then widen what this checks going forward. Totally fine. add automated security reviews and require every review thread resolve before merge. I love that they just drop this in here as though we can do this and our claude doesn't block us from asking questions about security stuff. There are solutions to this problem, but they're not going to be very frequent on the anthropic side. Sadly, match review efforts to risk. So, a quick pass on routine checks, a deep one where a missed bug is expensive, quick, standard, or deep encoder review. And then capping the review round so that you don't go off forever. like I've burned so many tokens with accidental forever reviews. Then you have the approval process. You write down risk tiers, yada yada. Set the risk based on the changes, not than the author. Keep lighter approval paths for authors you trust. Use code owners and branch protection. So O, billing, CI, and release flows, other important things. Get specific people. Never let a bot's approval replace the humans. Yes. Then we have the merge and after section. Use a merge Q, feature flags, internal use first, etc. Staged releases and onestep roll backs are huge as well. Turn incidents and reverts into eval sets.

Rescore the reviewer when it prompts and use the prompts, tools, and model changes to get a new score to figure out if the model's actually making progress here. Anything you cannot measure, you cannot fix at this point because AI is so good at fixing things when you give it the tools to climb against. And if you can make an eval of the types of changes that have been getting into your code that shouldn't have and turn that into a test you can run, it gets very easy to fix these things. I talk about this a lot in my hill climb video where I talk about running claude forever, I go in depth on how to set these types of environments up. They also suggest that measuring human time is useful with reverts and incident rates being measured as well. So, where is all of this going now? When they announced the code review product for Claude Code, they said that approving a PR is still a human call. The product posts findings, not approvals. Our own repos are now going further, letting owners decide which low-risk changes merge on automated checks alone, and that line keeps moving. When a review misses something, we ask what the model didn't have in front of it, and we put the fix in a cloud MD, a skill, or a new check. Nope, this is not right. This is the I do not like this guidance. You should try to fix the code first. And if you can't fix the code in a way that prevents the problem, then you add a check. If you can't fix it there, then you might maybe add a skill or a cloud MD. We somewhat recently overhauled all of our repo guidance. And I bet if we go to the agent, oh, we did make a change to agent MD eight hours ago. Let's see how frequently this file changes. More than it used to, but still not that much.

One month of changes still fits on one page, which is much less than other places in this codebase. The chat view component, for example, changes like 30 to 50 times a day. Our agent MD has changed onceish a dayish and even less earlier on. Like it changed once every fiveish days before. So like we're not constantly adjusting all of this. Did Julius move us off of TS Rust? Oh, Julius, you should have just told me to keep going. I would have just spun back up my agent. Apparently we turned off of TS Rust because Julius wanted new effect features. But yeah, like actual guidance change there though. We didn't adjust it because we saw some small thing that annoyed us. So like even there, we didn't change this because we saw our agents doing the wrong thing. We changed it because the repo itself changed and one of the pieces of guidance was no longer true. That is what leads us to changes, not problems in the codebase that we're trying to like give you like language to avoid.

Imagine that you're like blind and you're navigating a maze in a dark room and you keep running into walls and tripping. We could give you better instructions before you walk into the maze. Or we could just cover up the holes and make the guidance in the maze better so it doesn't matter how well you can see when you walk in. Like I don't want to just keep giving the model more instructions before it goes in the door. I want to make it so the room it's navigating in is as easy for it to navigate as possible. And I don't think more and more of these review MD files are the solution at all. They actually have a call out at the end here asking to hear more from us about which checks we're automating, which ones still need people, and what we think they have wrong. I'm even more excited about this video going out now cuz I know they'll listen. They might actually try some of these things, especially now that it's like Addie and Emma who are two old friends. I also like Emma's absolutely unbelievable. I'm so happy that she's at Anthropic now. Not as familiar with Michael and Amy, but I've heard good things. This is an awesome article. It's really nice to have Anthropic being this much more transparent with their process now. blogs like this just did not exist from them a year ago. They were not sharing this much about how they worked internally. And it's such a nice change to get this type of info. Hopefully, you enjoyed this as much as I did. I love peering behind the curtains on how people work with these tools. And maybe you got to learn some things about how you can use them, too. Let me know how you guys are doing reviews in the comments. And until next time, peace nerds.