Faster Engineers Are Not Yet Proof of a Faster Team

I have been thinking about how we measure the impact of AI on a software engineering team.

When we started building an AI harness around my team's workflow, most of the attention went, naturally, into making the process reliable: giving agents the right context, making expectations explicit, adding verification steps, and working out where human judgment still had to sit. As the harness evolved, a question we had not focused on as much kept coming back: how would we know whether any of this was making the team better?

It is tempting to look at how quickly work gets implemented, how many pull requests get merged, or how much time engineers feel they are saving. Those numbers tell us something. I am not convinced they tell us enough on their own. Making implementation faster does not mean the rest of delivery has become faster with it, and it says even less about whether what we ship is holding up.

In the post on moving from coding agents to software factories, I wrote that acceptance and rework say more than adoption does. That sentence was easy to write. Turning it into something a team can measure is harder, and it is where this series starts. My suspicion is that understanding whether AI makes a team faster requires looking well beyond what happens during implementation: at the whole path a change travels, from the moment someone decides it is worth doing to the moment it has been in production long enough to trust, and at the human attention that path now consumes.

Faster at what, exactly?

"Faster" hides at least three different things. There is the speed of one engineer producing a change. There is the speed at which changes get through review, integration and release. And there is whether what shipped stayed shipped and was worth shipping at all. For a long time those moved together closely enough that we could afford to be careless about the difference.

They seem to be coming apart, and CI data is one of the few places where you can watch it at scale, as long as you remember what it measures. CircleCI's 2026 State of Software Delivery, built on almost 29 million workflows from September 2025, found average daily workflow runs up 59% year on year. For the median team, though, feature-branch throughput rose 15% while main-branch throughput fell 7%. Its Q3 Pulse, published a couple of weeks ago, complicates the picture: main-branch success rates improved, from 70.8% in the first quarter to 78.3%, and recovery got a little quicker, yet the median main-branch workflow still runs 1.7 times a day, unchanged from the previous quarter.

A workflow run is not a deployment, and a green main branch is not a feature in a customer's hands. CircleCI describes main-branch throughput as an indicator of how often teams integrate code, and a rough one at that: more work is being started and validated, without a matching rise in how often it gets integrated into main. What reaches customers is another question the data cannot answer.

The individual experience points the other way, which is part of what makes this confusing. In Atlassian's State of Developer Experience 2026, 87% of developers say AI helps them ship faster. In the same survey, the share reporting ten or more hours saved a week fell from 68% in 2025 to 25%, with a smaller sample than the year before and no single cause the report can isolate. Feeling faster is real data, and I would not throw it out. But it is data about how the work feels, and not about how the work flows.

Why measuring AI productivity is harder than it looks

If anyone was going to answer the individual question cleanly, it was METR. Its 2025 randomized study found that experienced open-source developers took 19% longer on tasks where AI was allowed, while believing afterwards that it had sped them up by around 20%. That result travelled far, mostly stripped of its caveats.

The update METR published in February 2026 is more useful than the original finding, at least for anyone trying to measure their own team. The follow-up study started in August 2025 with 57 developers and more than 800 tasks, and it did not produce a reliable estimate. Between 30% and 50% of the developers said they were choosing not to submit some tasks because they did not want to do them without AI, so the tasks with the most to gain were quietly leaving the sample. Developers also found it hard to account for their time when they were working on something else while agents ran in parallel, which is the same kind of attention most delivery metrics never see. METR described its data as very weak evidence for how large the speedup now is, and noted that the true speedup could be much higher among the developers and tasks that had selected themselves out.

What stays with me is the reason the number could not be trusted. A research team with randomization and paid participants could not hold the comparison still, because the tool had already changed which work people were willing to do and how they spent their time while doing it. If your evidence is last year's sprint report pasted into a spreadsheet next to this quarter's, you are dealing with the same problem without the randomization.

And the problem is not only statistical. Before and after are rarely the same team doing the same work: the mix of tasks shifts, people join and leave, a reorg lands mid-quarter. It is better to know where that noise comes from than to pretend it is not there.

The costs we do not always see

Some costs appear immediately. Others only become visible months after the change has shipped, and those are the ones a quarterly chart is worst at catching.

The 2025 DORA report, drawing on nearly 5,000 respondents, found AI adoption positively associated with delivery throughput, unlike the year before, and still negatively associated with delivery stability. It is a survey-based association, not proof that AI causes instability, but it is still the reason DORA's metrics are worth reading as a set rather than one at a time. DORA groups its five metrics into throughput and instability. Change lead time and deployment frequency, two of the throughput ones, describe how quickly committed changes move through the delivery system; change fail rate and rework rate describe what that pace may have cost. Neither is a literal account of a feature's journey from idea to customer. Looking at any of these metrics in isolation can hide improvements or problems elsewhere in the delivery process.

The slower costs are harder to see. The CMU study of Cursor adoption, published at MSR 2026 as Speed at the Cost of Quality, compared open-source projects that adopted Cursor with matched projects that did not. The design is quasi-experimental, and open-source repositories are not product teams with on-call rotations. Within those limits, the authors found a large but temporary rise in velocity, and a persistent increase in static analysis warnings and code complexity that they identify as a major factor in the slowdown that followed. Faros AI's 2026 engineering report describes something similar in telemetry from some 22,000 developers, with throughput up while incidents per pull request and time in review rose much faster. Faros sells measurement, and the data is correlational, so I would read it as a direction, not a set of figures.

Then there is the cost that rarely appears in any report: the people. Review is often where part of the speed gained during implementation goes. A pull request records when it was opened and when it was merged, not the time someone spent reading the diff twice, running it locally, rewriting a test the agent got subtly wrong, or the afternoon they had planned for something else. Atlassian's full report found that 65% of developers regularly spend time correcting AI output that missed the original intent, and close to half spend four or more hours a week managing AI tools.

The emotional side of that, the human latency that AI-accelerated cycle times leave behind, already has a post of its own. What matters here is the bookkeeping. Supervision is a cost, and a cost nobody counts does not disappear. In practice it tends to land on the most experienced engineers on the team.

What we would actually measure

Before choosing any metric, a team has to decide what it would consider an improvement, and which costs it is willing to accept for it. That sounds obvious. It is also the step most measurement efforts skip.

It helps to separate three questions that tend to get blended into one. The first is about activity: are we producing more changes, or finishing tasks faster? The second is about delivery: are useful changes reaching production more often, without stability getting worse? The third is about sustainability: what does it cost, in people, money and operations, to keep that pace? Many of the charts that circulate answer the first and get presented as an answer to the second. The third is rarely asked.

The answers also depend on the work. A team using agents to write documentation, to implement product features, to work through incidents or to refactor a complex system is doing four different things, with different metrics and different tolerances for risk. Imagine a team whose lead time improves by a fifth while its change fail rate creeps up slightly. Is that better? For an internal tool where a failed deployment costs an hour of someone's afternoon, probably. For a payments flow, probably not. The numbers cannot make that call. They can only make it visible.

What not to count is clearer. Atlassian found that 60% of developers say their organization treats AI use as a productivity signal, 41% say they sometimes use AI more than necessary because they feel expected to, and only 42% of leaders believe they have the metrics to tell real gains from activity. Usage, lines of code and pull request counts were always weak proxies, and they get weaker when the tool being evaluated can produce them on demand. In the Cursor study, lines added almost quadrupled in the first month after adoption and had faded after the second. When a metric becomes easier to influence by using the tool, it gets harder to know whether we are measuring an improvement or encouraging a particular behaviour.

A first attempt does not need to be a programme. Something small enough to finish in a quarter, and honest enough that the result could disappoint you, will do. Most of the work happens before the rollout:

• A baseline, recorded first, for the stream of work the agents will touch: deployment frequency, change fail rate and rework rate, plus an end-to-end lead time from "ready to start" to "in production", defined for the experiment and wider than DORA's change lead time, which starts at commit. Both are useful, as long as nobody confuses them.

• What worse would look like, agreed in advance: the movement in stability, review time or incidents that would make you slow down, written down before anyone has seen the numbers they hoped for.

• The human cost alongside: time in review per change, how often someone has to step in during an agent run, and a short time-use check-in each month. That check-in is also where opportunity cost shows up: if a senior engineer spends more of the week reviewing agent output, something else is not getting done, whether a design conversation, time with a newer colleague, or the system only they understand.

• A comparison that is not only before and after. If two comparable teams or services exist, stagger the rollout so one can act as a rough control for the other. If not, keep a log of everything else that changed: the reorg, the holiday month, the incident that ate a sprint.

• A lagging look at quality: complexity trends and incidents traced back to agent-assisted changes, revisited after a few months rather than at the end of the first exciting one.

Cost belongs in the picture too, though less neatly than it seems. Model spend arrives on an invoice; review hours and incidents are spread across everyone's calendar. Dividing the total by the changes that reached production and stayed there can be a useful complementary signal, as long as you remember what it hides. Changes differ in size and in value, and counting only the ones that survived leaves out the discarded attempts, which are part of the cost. DORA's ROI of AI-assisted Software Development report, as quoted in InfoQ's coverage, calls its own figures "a high-uncertainty estimate meant to spark a conversation", and argues that the main burden has shifted from inference to governance, workflow change and upskilling. That humility seems right for anyone doing this at team level.

The same report expects a dip before the gains, which raises a fair question: how long do you wait before deciding whether an adoption is working? The report warns that leaders who read the dip as failure may pull out too early. But once everyone expects a dip, almost any bad quarter can be explained away. So a team has to choose how long it will wait, and ideally choose it before the first bad quarter arrives.

And then the limits. A small team will rarely be able to demonstrate a modest improvement with much confidence. What it can do is notice large changes, catch problems early and make better decisions about what to try next. That is a reasonable goal for a first quarter. And more honest than a precise-looking improvement figure nobody can reproduce. The rest comes from asking the people doing the reviews, as a standing question in the retrospective: where did the time go this month that no chart shows?

Final thoughts

Measuring whether AI is making a team faster will rarely produce a clean answer. What most teams will get is a range, a couple of trade-offs they can name, and a list of things they still cannot see. Even the strongest studies here describe associations in particular contexts, and a team's own comparison will be noisier than any of them.

I want these workflows to work. I have spent enough time building around coding agents to see how much they can help, and I do not think we need to pretend otherwise. Which is why I would like to know where the improvement comes from, and what we might be overlooking on the way. I doubt there is one number that will prove the investment was worth it. What measuring can do is make the changes in how we work visible enough that we can decide, together, whether they are taking us somewhere good.

It is a smaller thing than an answer, and probably a more useful one: sharper questions about what is improving, what it costs, and whether the trade-offs make sense for the people doing the work.

And even a team that measures all of this well will still be measuring delivery, which is not the same thing as the business getting better. That is a different gap, and nothing in a delivery pipeline will show it.