There's a great new paper published by some Harvard PhD candidates, and the assertion is that while AI generates more code, firms are not producing more software. The theory is that gains made in coding are lost in reviewing. To me these are two parts to the same thing, but the conclusion points out that a portion of the SDLC viewed in isolation does not itself measure the entire outcome.
I have some thoughts.
First off, more code as a measure of anything wasn't a good benchmark for humans, and it's not for robots either. Ditto for pull requests. Anecdotally, my observation is that PR's are bigger than they've ever been, so if reviewing is the bottleneck, perhaps it's because the work tasked to the robots is too large. If you strictly view the LLM's as a means to generate code faster than you can write it, then you need to have an apples/apples comparison. Give it stuff in the same chunks you would do, and compare to that. For folks wanting to throw it over the wall and expect complete software the next day, that's the equivalent of giving some requirements to a developer and saying, "Come back when you're done." That's never worked for people, because the refinement, feedback and discovery loop is what makes great software.
If you strictly adhere to this human-like approach, logically, the time for code review does not change. And since you saved time (not) writing the code, you have more time to review. But you don't need it if the work was scoped the same way you would do it for people. The review is then not a bottleneck. You've saved the time. If you've over-scoped the work, yes, of course it'll take longer to review. Scope control is more important than ever if you want to actually achieve faster shipping. I'm aware that you put the agents on reviewing as well, but there's a reason we don't have engineers review their own code. Why would we treat the robots differently?
Someone asked me, "What is the next bottleneck, and how do you deal with it?" This is a question fallacy. It's predicated on the idea that there is a bottleneck caused by the shift in production, and that it can be solved with process or AI. Coding, if I were to guess, was never more than 20% of total human hours in the SDLC, end-to-end. Most of it is humans deciding the outcomes they're after, debate about how to solve those problems and validation with users that the outcomes are arriving as intended. You can't automate that, because LLM's are word guessers that have no wisdom or judgment, let alone the ability to read your mind, or the minds of entire teams. You build something because someone decided on a business outcome. That decision making is the time consuming part.
If you control the company purse, you might be asking, "Why do we even spend this money if the outcomes don't change?" It's a valid question. For reasons unclear to me, engineers are focusing on how to use AI to automate more, use more agents, and spec out as much as possible up front. It's unclear why. The goal is not token-maxing or spending more. The goal is still to ship the best software possible. More agents writing more code does not inherently lead to better software.
When we continue to work the way we always have, not reinvent the wheel, deferring smaller chunks of work to the robots, we save time and ship faster. Even if we save 20% in coding time, it's like getting back a day a week to do something else. With that saved time, we can shore up our software by tackling the tech debt we would otherwise ignore. We can prototype in a meaningful way that results in some usable bits. We can stop letting our software suck.
No comments yet.