Development in the age of AI...some metrics

Intro

So, you've all seen the hype. You've maybe experienced how awesome (and highly addictive!) Claude Code feels.

From an enterprise perspective, in an engineering team of ~30 active engineers committing to well over ~40 active repos, what's it meant from a data perspective over the last 3 months?

Well....here's the highlights:

• I'm pleased to announced that change failure rate has remained steady.
• PR review time has remained steady on average.
• PR Size (average files per commit) has gone up. ~6 files on average to ~11 files on average.
• Review engagement (number of reviewers per PR) has dropped slightly (1.8 to 1.4 on average)
• The most striking stat is that - per week - 3 times the number of files are changing (addidtions, edits, deletions) now than were changing before the mass introduction of Claude Code.

So, what does all this mean? Here's some potential inferences...

• People are able to do more work with AI?
• People are happier to trust an AI to review their code?
• AI is generating more code than it needs to?
• AI is touching more files than it needs to?
• We're doing larger features in bigger batches?

It reassuring to see that our change failure rate has remained steady. The engineering team is experienced and we're clearly still creating the right code, rather than the wrong code but faster.

What's next?

As more departments adopt Claude as part of their day to day, we're seeing increasing numbers of working POCs, features that would usually be considered too hard to achieve without AI added into the roadmap.

We're increasingly seeing numbers of non-engineering users creating solutions to their unique problems. This is not without risk, but that's a blog post for another day.

As an Engineering Director I'm keen to see where we can take this next. I still truly believe software engineering on an enterprise level is a team sport. What I'm not witnessing is that teams are collaborating with AI together - we're seeing lots of individual efforts within teams, but not team efforts (if that makes sense?). I think this is due to there not being an easy way to do this as yet (more pairing etc is one way).

There's no easy way to share context, plans and preparation across a broad spectrum of people. We're still logging work that's broken down to the ticket level - maybe this needs to change? There's work to do on standardising artefacts checked-in to source control. Is it worth checking in plans for example?

Non-technical people are realising that domain knowledge, expertise and experience are massively valuable in this new age. Getting this context loaded up in such a way that engineers can tap into it will be a game changer.