I’ve seen a few posts on social media with questions like:
Why are we still trying to figure out exactly how much more productive developers are with GitHub Copilot when it is so cheap? Who cares? Just give them the tool!
Well, let’s think about that. Who does care? (And is Copilot cheap?)
I’m imagining a conversation like this:
Person A: It will make [or it made] our developers more productive!
Person B: How much more productive?
As my dad used to say, “what are you going to do with that information?” If you’re not going to act differently based on what you learn, is the effort to find out going to be worth it?
Person B, what are you going to do if the answer is “10% more productive” vs. “50% more productive” vs. “300% more productive”?
Just curious
Sometimes my answer to my dad was: “I was just wondering.” Asking a question can still be worth the effort, even if you’re not going to do anything at all with the information.
In this case, though, Person B: is it worth someone else’s effort just to satisfy your curiosity? Seems iffy to me.
Purely research
Maybe Person B is a researcher. Are they comparing different tools to see which is more effective? Are they collecting a baseline to monitor changes over time as tools (hopefully) improve? Are they just trying to be precise, because “I dunno, it’s more than a few percentage points” isn’t great in a scholarly article?
Persuading people to use the tool
Could be that Person B is trying to convince someone to adopt Copilot. They are selling licenses. They think it looks interesting and want access for their teams. They bought a bunch of licenses already and want them to go to good use. They think AI is the way of the future and want to make sure people are on board. And they are hoping for a convincing-sounding number.
Return on investment (ROI)
Perhaps Person B has been pushing Copilot because they thought it was cool (or because their management told them to push Copilot), and now management wants to know if this was a good use of resources.
Could be some judgment involved here too. Person B might be checking for confirmation of their suspicions, thinking “I bet it’s not great, AI isn’t that big a deal,” or “I’d say anything less than ‘2x as productive’ and they’re using it wrong.”
Speaking of resources, is it actually cheap?
People say, “it’s only $20” (per user, per month). True, but there’s also developer time spent learning and experimenting. Add to that, potentially, costs like training, governance, monitoring, reporting, legal considerations, privacy and security concerns, risk management, etc.
Maybe that’s still cheap compared to how expensive developers are, but it’s not quite as simple as “if I cost $50/hr and it can save me an hour a month, then I should have the tool.” That simple math falls through when your developer saves a few hours but also inadvertently exposes private data to the public, causes the company to lose millions in some transactions gone awry, or introduces a security vulnerability. And the company is held accountable by auditors who say there should have been more training.
Changes of plan
My first thought when I tried to answer the question of why find the specific percentage: are we doing math? Like “if developers are 50% more productive, that means every person can now do the work of 1.5 people. That means 2 people with AI equals 3 people without AI. So I can reduce my headcount by 1/3!”
Or: “if developers are 50% more productive, then what they used to get done in 3 weeks should now take only 2 weeks. So I can move up my delivery date projections!”
Both of these worry me…
So many questions
The more I write, the more this topic is raising other related questions I want to talk about. I’m sure I’m omitting plenty of other reasons for seeking a specific percentage. But some of the questions raised:
- How do we define productivity? A classic, and a great question to ask anytime that word starts getting tossed around.
- How did we get this 50% number? Is it even real? (Okay, in this newsletter it’s for illustration purposes. Please don’t cite my 50% as fact, okay? I made it up. But in real life, if someone gives you a number, ask where it came from and what it actually represents.)
- Why the math above to reduce headcount might be a problem
- Why the math above to move delivery dates up might be a problem
- What skills do developers with AI need in order to get the biggest productivity gains?
I’m sure I’ll be writing more from the questions sparked by this post. I’m still sorting out what writing belongs in my blog vs. what belongs in my newsletter (sign up here!), but I already have something I want to write next. I want to start in exactly the same place as this blog post, but then take it in a completely different direction. That will probably be in the newsletter in the next week or two.
Make sure to join my email mailing list so you are getting the goods right in your inbox. In the meantime, drop a note below with your thoughts about AI and developer productivity. Have you been asked to give a percentage? How did you measure it?
