Another post about AI? I just made one earlier this afternooooon…
But then Forrest Brazeal posted this, which got me thinking:
If we treat AI like a junior developer, do we interview AI the way we interview junior developers? Make AI do a whiteboard coding exercise? Insist that it can demonstrate more years of experience with your tech stack than are appropriate for a junior level? Put it through several rounds of interviews and then ghost it?
Walk me through your thought process here. Photo by Walls.io on Unsplash
I’ve never heard of anyone doing that. Why not? Probably because hiring a human developer is a Big Deal involving great expense, and something like an AI coding assistant is a comparably smaller expense.
I wonder if there’s also a measure of “with humans, we need to consider what this person is like as a colleague,” with a corresponding assumption that we don’t need to consider that for AI because it’s not a person. Is that true?
Granted, AI will get its fingers all over your codebase in a way that no single developer can, because you’ll deploy it on every team. But that’s okay, because just like hiring an army of junior developers, you’d have senior developers providing careful oversight. Hm. Is that true now?
I wonder who’s easier to “let go,” if they aren’t working out: AI, or human. Who is evaluated more carefully. What “working out” means for a developer, anyway. Oh, we’re back to the topic of “what makes an effective/productive developer” again.
I promise my newsletter isn’t all about AI. Sign up below!
I’ve seen a few posts on social media with questions like:
Why are we still trying to figure out exactly how much more productive developers are with GitHub Copilot when it is so cheap? Who cares? Just give them the tool!
Well, let’s think about that. Who does care? (And is Copilot cheap?)
I’m imagining a conversation like this:
Person A: It will make [or it made] our developers more productive!
Person B: How much more productive?
As my dad used to say, “what are you going to do with that information?” If you’re not going to act differently based on what you learn, is the effort to find out going to be worth it?
Person B, what are you going to do if the answer is “10% more productive” vs. “50% more productive” vs. “300% more productive”?
Sometimes my answer to my dad was: “I was just wondering.” Asking a question can still be worth the effort, even if you’re not going to do anything at all with the information.
In this case, though, Person B: is it worth someone else’s effort just to satisfy your curiosity? Seems iffy to me.
Purely research
Maybe Person B is a researcher. Are they comparing different tools to see which is more effective? Are they collecting a baseline to monitor changes over time as tools (hopefully) improve? Are they just trying to be precise, because “I dunno, it’s more than a few percentage points” isn’t great in a scholarly article?
Persuading people to use the tool
Could be that Person B is trying to convince someone to adopt Copilot. They are selling licenses. They think it looks interesting and want access for their teams. They bought a bunch of licenses already and want them to go to good use. They think AI is the way of the future and want to make sure people are on board. And they are hoping for a convincing-sounding number.
Return on investment (ROI)
Perhaps Person B has been pushing Copilot because they thought it was cool (or because their management told them to push Copilot), and now management wants to know if this was a good use of resources.
Could be some judgment involved here too. Person B might be checking for confirmation of their suspicions, thinking “I bet it’s not great, AI isn’t that big a deal,” or “I’d say anything less than ‘2x as productive’ and they’re using it wrong.”
Speaking of resources, is it actually cheap?
People say, “it’s only $20” (per user, per month). True, but there’s also developer time spent learning and experimenting. Add to that, potentially, costs like training, governance, monitoring, reporting, legal considerations, privacy and security concerns, risk management, etc.
Maybe that’s still cheap compared to how expensive developers are, but it’s not quite as simple as “if I cost $50/hr and it can save me an hour a month, then I should have the tool.” That simple math falls through when your developer saves a few hours but also inadvertently exposes private data to the public, causes the company to lose millions in some transactions gone awry, or introduces a security vulnerability. And the company is held accountable by auditors who say there should have been more training.
Changes of plan
My first thought when I tried to answer the question of why find the specific percentage: are we doing math? Like “if developers are 50% more productive, that means every person can now do the work of 1.5 people. That means 2 people with AI equals 3 people without AI. So I can reduce my headcount by 1/3!”
Or: “if developers are 50% more productive, then what they used to get done in 3 weeks should now take only 2 weeks. So I can move up my delivery date projections!”
Both of these worry me…
So many questions
The more I write, the more this topic is raising other related questions I want to talk about. I’m sure I’m omitting plenty of other reasons for seeking a specific percentage. But some of the questions raised:
How do we define productivity? A classic, and a great question to ask anytime that word starts getting tossed around.
How did we get this 50% number? Is it even real? (Okay, in this newsletter it’s for illustration purposes. Please don’t cite my 50% as fact, okay? I made it up. But in real life, if someone gives you a number, ask where it came from and what it actually represents.)
Why the math above to reduce headcount might be a problem
Why the math above to move delivery dates up might be a problem
What skills do developers with AI need in order to get the biggest productivity gains?
I’m sure I’ll be writing more from the questions sparked by this post. I’m still sorting out what writing belongs in my blog vs. what belongs in my newsletter (sign up here!), but I already have something I want to write next. I want to start in exactly the same place as this blog post, but then take it in a completely different direction. That will probably be in the newsletter in the next week or two.
Make sure to join my email mailing list so you are getting the goods right in your inbox. In the meantime, drop a note below with your thoughts about AI and developer productivity. Have you been asked to give a percentage? How did you measure it?
Another post I made a while ago on Medium, but this one I updated today with some newer thoughts.
I once had a colleague who (jokingly) left this comment on a code review, and not in reference to a specific line of code:
“Missing semicolon.”
Photo by Nubelson Fernandes on Unsplash
Case study two
I once had a QA colleague who reported a bug to me in some code I had just written.
I made a code fix, but he said it was still broken.
Another fix, still broken.
Another, still broken.
Then he messaged me: “Never mind! User error!” and explained what he’d done wrong.
I think of both of these case studies often.
This post originally ended here, with “Yeah, that’s the whole post :)” when I posted it as “Ghost errors” on Medium on 24 August 2023. But I’m posting it again in 2025, and there’s more to say.
First of all…
What the above case studies have in common: reviewing something again, even if you’re not entirely sure what you’re looking for, can be a good thing.
My reviewer colleague was joking about the missing semicolon. I think. Maybe I should just look through the code one more time.
And my QA colleague was not joking, but the “bug” he was reporting was not actually a malfunction of the software (although it was arguably a design issue if even someone so knowledgeable about the product would be using it “wrong”). Regardless, it led to my finding and fixing three other problems that neither of us realized were there.
Case study three
A colleague recently was working on an application that had a bug that only happened at a certain time of day. He asked his AI coding assistant tool to write him a set of tests to test all the places in the code where time of day was a factor.
And one of those tests failed.
Questions
Have you ever gone into your code to look for a bug that wasn’t actually there – but in the process, you’ve found some other things that need fixing?
Wonder what happens if we send our AI assistants into our code to identify the bugs for us? Have you tried this? I’ve tried asking my coding assistant for suggestions on improving the code, but I haven’t said “there are some bugs in this code, please identify them and suggest fixes.”
And here’s the question most on my mind right now:
Do we lose something in using the AI tools? Is this the equivalent of using a dirt mover instead of using shovels – more powerful, less toil but also less exercise, probably better in some circumstances? What am I doing to keep myself in shape in case I need to pick up a (code debugging) “shovel” someday?
