Category: AI

  • Vibe coding made less terrifying

    Vibe coding made less terrifying

    It’s been a long time since I’ve felt I like I was hanging on every word of a book.

    I was so done reading about AI. Then I saw that IT Revolution was publishing a new book:

    Vibe Coding: Building Production-Grade Software With GenAI, Chat, Agents, and Beyond, by Gene Kim and Steve Yegge.

    The book isn’t out until October, but I have a few sample chapters of the ebook because I’m on IT Revolution’s mailing list. And I realized that, despite my current dread of all things AI-related, I should read those sample chapters before attending the seminar.

    Wow, y’all. It’s been a long time since I’ve felt like I was hanging on every word of a book. This is one of those books.

    I’ve already added it to my list of books I recommend. And I have a whole list of people, developers and non-developers alike, who I think would enjoy it. (I may be able to get you a discount. More about that later.)

    Skeptical of vibe coding? I sure was, and to some extent still am. Gene and Steve were, too.

    But they are clearly converts, and the book is getting me to rethink my initial impressions.

    Addressing concerns

    Yes, there are serious concerns about writing “real” production code with vibe coding, where you’re not really delving into the code output.

    • Is the coding process itself safe?
    • Is the code it generates secure, maintainable, and extendable?
    • If you don’t understand or even look at the code, how can you be sure there isn’t some terrible problem lurking that will cause major problems once it’s in production? Okay, you can never be sure of that, but it seems so much worse if you don’t fully understand the code.

    It’s clear that the authors of this book are grappling with these same concerns. The chapters where they talk about how to approach vibe coding for better results are still ahead, and not part of the sneak preview! Argh, cliffhanger! 🙂

    Magic and hope

    One of my early takeaways isn’t directly related to vibe coding at all: I finally understood why, after 20 years as a developer, I just didn’t want to write code anymore. The tedious parts have started to outweigh the magic. If you’re feeling that way, too, you’re not alone.

    Vibe coding may be a way for developers to rediscover that magic. It may also be an opportunity for people not trained as developers to discover it too, and to create new software directly.

    The book also gives me hope that the role of the developer, even the junior developer, isn’t disappearing. Changing, yes, that’s inevitable. But they point to other historical moments where people feared the end of our profession, and those moments turned out to be times of growth.

    Learning more

    My thoughts these days are all about the role of humanity for software developers, so I’m interested to see what I’ll learn from Gene and Steve’s session and the conference itself. I’ll report back on my mailing list after the conference.

    Does this Vibe Coding book sound like a good read to you, too? Want 15% off a paperback copy?

    If there’s interest, I will coordinate a bulk order for 15% off for folks on my mailing list. Email me or message me on LinkedIn by 9/28 to let me know if you’re interested.

    Join the mailing list below so you don’t miss a thing!

  • Did I chat with an AI?

    Did I chat with an AI?

    I try to avoid posting about AI, but today I found myself wondering: did I just chat with an AI?

    Trying to decide which of several products to purchase online, I used the company’s website chatbot, which transferred me to “Vanessa” for further assistance.

    Was Vanessa real? I found myself evaluating every line.

    The overly-enthusiastic and strangely polished parts? That’s probably AI.

    But what about…

    • The natural-sounding, competent, useful responses?
    • The grammatical errors?
    • The inability to remember bits of context from earlier in the conversation?
    • Delays where the “agent is typing” and I’m waiting…
    • …but then I get an “are you still there?” a minute later while I’m typing?
    • The occasional complete nonsense, sort of an on-topic word salad?

    What’s wild to me is that I have experienced all of those things in the days before AI chat, when you could be reasonably certain you were talking to a human. So, on one hand, any of these could be a human behind the wheel.

    A human hand holding a robot hand as if in a handshake
    As if AI chatbots had a physical incarnation with a hand

    But then again, maybe AI would say: “How is you’re day so far?” If so, was the grammar error deliberately inserted, to make it sound more human? Or did AI just learn from a lot of grammatically incorrect training data?

    Humans of yore

    Once upon a time, you chatted (phone or text) with a human. They either knew how to help, or they would have you hold while they talked to someone else on your behalf. You could picture someone sitting in an office somewhere, spinning their office chair around to ask the senior person at the cubicle behind them.

    Later, you chatted with a human who generally didn’t know how to help, but who could follow a flowchart of common questions. I worked in tech support, I get it. 90% of the calls are answered by the flowchart.

    Unfortunately, those people would often not recognize when you were off the flowchart. I might say “step 3 of your instructions refers to a button that doesn’t exist,” and they’d just send me the link to the same instructions I was already following.

    When they ran out of flowchart, they’d send you to tier 2. But even the flowchart person was still someone sitting in an office somewhere.

    “The button doesn’t exist? That’s not on the flowchart.” Photo by Arlington Research on Unsplash

    They might be following a script. But when it said “Agent is typing…” on the screen, they were at least assembling a reply from various bits of canned text with a little original text thrown in.

    Where’s my person?

    It was never clear to me how much “Vanessa” was controlled by a human operator. If at all.

    I hope for the sake of that company that there was at least some human oversight to prevent inappropriate responses.

    But, at best, I imagined someone monitoring multiple chats to make sure the AI responses weren’t going off the rails. Would they even need to know the product? Maybe their software could be doing some realtime sentiment rating to highlight if the user is getting frustrated.

    The contrast was striking when, later in the day, I had a chat with “Julian” from Microsoft’s online sales chat feature. Aside from one overly enthusiastic greeting (on the order of “It’s an absolute delight to assist you today! How can I help you make your business more successful?”) the conversation… just felt normal.

    The wording and grammar weren’t perfect. But they didn’t seem unprofessional, they just seemed human.

    Microsoft likely has access to the best AI a company can get. If I had to pick a company that might have AI so good they could fake me out, they’d certainly be on my list of candidates.

    But I feel like Vanessa’s response to “I honestly could not tell if I was talking to a real person or an AI” might have been something more… AI-sounding.

    What would an AI say if asked that, anyway?

    I couldn’t get claude.ai to be deceptive about it, even hypothetically. But I did get it to tell me what a human could say to reassure. I wanted an AI-generated example to contrast with Julian’s response above.

    “Ha, I get that question sometimes! I do use assistance with my writing to make sure I’m being as helpful as possible, but there’s definitely a human behind these messages.”

    Well done, Claude. No need to escalate to tier 2 support.

  • Interview with an AI

    Interview with an AI

    Another post about AI? I just made one earlier this afternooooon…

    But then Forrest Brazeal posted this, which got me thinking:

    If we treat AI like a junior developer, do we interview AI the way we interview junior developers? Make AI do a whiteboard coding exercise? Insist that it can demonstrate more years of experience with your tech stack than are appropriate for a junior level? Put it through several rounds of interviews and then ghost it?

    Walk me through your thought process here. Photo by Walls.io on Unsplash

    I’ve never heard of anyone doing that. Why not? Probably because hiring a human developer is a Big Deal involving great expense, and something like an AI coding assistant is a comparably smaller expense.

    I wonder if there’s also a measure of “with humans, we need to consider what this person is like as a colleague,” with a corresponding assumption that we don’t need to consider that for AI because it’s not a person. Is that true?

    Granted, AI will get its fingers all over your codebase in a way that no single developer can, because you’ll deploy it on every team. But that’s okay, because just like hiring an army of junior developers, you’d have senior developers providing careful oversight. Hm. Is that true now?

    I wonder who’s easier to “let go,” if they aren’t working out: AI, or human. Who is evaluated more carefully. What “working out” means for a developer, anyway. Oh, we’re back to the topic of “what makes an effective/productive developer” again.


    I promise my newsletter isn’t all about AI. Sign up below!

  • Exactly how much more productive?

    Exactly how much more productive?

    I’ve seen a few posts on social media with questions like:

    Why are we still trying to figure out exactly how much more productive developers are with GitHub Copilot when it is so cheap? Who cares? Just give them the tool!

    Well, let’s think about that. Who does care? (And is Copilot cheap?)

    I’m imagining a conversation like this:

    Person A: It will make [or it made] our developers more productive!

    Person B: How much more productive?

    As my dad used to say, “what are you going to do with that information?” If you’re not going to act differently based on what you learn, is the effort to find out going to be worth it?

    Person B, what are you going to do if the answer is “10% more productive” vs. “50% more productive” vs. “300% more productive”?

    Photo by kris on Unsplash

    Just curious

    Sometimes my answer to my dad was: “I was just wondering.” Asking a question can still be worth the effort, even if you’re not going to do anything at all with the information.

    In this case, though, Person B: is it worth someone else’s effort just to satisfy your curiosity? Seems iffy to me.

    Purely research

    Maybe Person B is a researcher. Are they comparing different tools to see which is more effective? Are they collecting a baseline to monitor changes over time as tools (hopefully) improve? Are they just trying to be precise, because “I dunno, it’s more than a few percentage points” isn’t great in a scholarly article?

    Persuading people to use the tool

    Could be that Person B is trying to convince someone to adopt Copilot. They are selling licenses. They think it looks interesting and want access for their teams. They bought a bunch of licenses already and want them to go to good use. They think AI is the way of the future and want to make sure people are on board. And they are hoping for a convincing-sounding number.

    Return on investment (ROI)

    Perhaps Person B has been pushing Copilot because they thought it was cool (or because their management told them to push Copilot), and now management wants to know if this was a good use of resources.

    Could be some judgment involved here too. Person B might be checking for confirmation of their suspicions, thinking “I bet it’s not great, AI isn’t that big a deal,” or “I’d say anything less than ‘2x as productive’ and they’re using it wrong.”

    Speaking of resources, is it actually cheap?

    People say, “it’s only $20” (per user, per month). True, but there’s also developer time spent learning and experimenting. Add to that, potentially, costs like training, governance, monitoring, reporting, legal considerations, privacy and security concerns, risk management, etc.

    Maybe that’s still cheap compared to how expensive developers are, but it’s not quite as simple as “if I cost $50/hr and it can save me an hour a month, then I should have the tool.” That simple math falls through when your developer saves a few hours but also inadvertently exposes private data to the public, causes the company to lose millions in some transactions gone awry, or introduces a security vulnerability. And the company is held accountable by auditors who say there should have been more training.

    Changes of plan

    My first thought when I tried to answer the question of why find the specific percentage: are we doing math? Like “if developers are 50% more productive, that means every person can now do the work of 1.5 people. That means 2 people with AI equals 3 people without AI. So I can reduce my headcount by 1/3!”

    Or: “if developers are 50% more productive, then what they used to get done in 3 weeks should now take only 2 weeks. So I can move up my delivery date projections!”

    Both of these worry me…

    So many questions

    The more I write, the more this topic is raising other related questions I want to talk about. I’m sure I’m omitting plenty of other reasons for seeking a specific percentage. But some of the questions raised:

    • How do we define productivity? A classic, and a great question to ask anytime that word starts getting tossed around.
    • How did we get this 50% number? Is it even real? (Okay, in this newsletter it’s for illustration purposes. Please don’t cite my 50% as fact, okay? I made it up. But in real life, if someone gives you a number, ask where it came from and what it actually represents.)
    • Why the math above to reduce headcount might be a problem
    • Why the math above to move delivery dates up might be a problem
    • What skills do developers with AI need in order to get the biggest productivity gains?

    I’m sure I’ll be writing more from the questions sparked by this post. I’m still sorting out what writing belongs in my blog vs. what belongs in my newsletter (sign up here!), but I already have something I want to write next. I want to start in exactly the same place as this blog post, but then take it in a completely different direction. That will probably be in the newsletter in the next week or two.


    Make sure to join my email mailing list so you are getting the goods right in your inbox. In the meantime, drop a note below with your thoughts about AI and developer productivity. Have you been asked to give a percentage? How did you measure it?

  • Bug hunting

    Another post I made a while ago on Medium, but this one I updated today with some newer thoughts.


    I once had a colleague who (jokingly) left this comment on a code review, and not in reference to a specific line of code:

    “Missing semicolon.”

    Photo by Nubelson Fernandes on Unsplash

    Case study two

    I once had a QA colleague who reported a bug to me in some code I had just written.

    I made a code fix, but he said it was still broken.

    Another fix, still broken.

    Another, still broken.

    Then he messaged me: “Never mind! User error!” and explained what he’d done wrong.

    I think of both of these case studies often.

    This post originally ended here, with “Yeah, that’s the whole post :)” when I posted it as “Ghost errors” on Medium on 24 August 2023. But I’m posting it again in 2025, and there’s more to say.

    First of all…

    What the above case studies have in common: reviewing something again, even if you’re not entirely sure what you’re looking for, can be a good thing.

    My reviewer colleague was joking about the missing semicolon. I think. Maybe I should just look through the code one more time.

    And my QA colleague was not joking, but the “bug” he was reporting was not actually a malfunction of the software (although it was arguably a design issue if even someone so knowledgeable about the product would be using it “wrong”). Regardless, it led to my finding and fixing three other problems that neither of us realized were there.

    Case study three

    A colleague recently was working on an application that had a bug that only happened at a certain time of day. He asked his AI coding assistant tool to write him a set of tests to test all the places in the code where time of day was a factor.

    And one of those tests failed.

    Questions

    Have you ever gone into your code to look for a bug that wasn’t actually there – but in the process, you’ve found some other things that need fixing?

    Wonder what happens if we send our AI assistants into our code to identify the bugs for us? Have you tried this? I’ve tried asking my coding assistant for suggestions on improving the code, but I haven’t said “there are some bugs in this code, please identify them and suggest fixes.”

    And here’s the question most on my mind right now:

    Do we lose something in using the AI tools? Is this the equivalent of using a dirt mover instead of using shovels – more powerful, less toil but also less exercise, probably better in some circumstances? What am I doing to keep myself in shape in case I need to pick up a (code debugging) “shovel” someday?