What Building Real Software With AI Actually Taught Me

I spent the last several months shipping production code with an AI doing most of the typing. I still read every diff and signed off on every line. Here is what that actually taught me, the part that doesn’t make it into anyone’s demo.

Confidence Is Not a Signal of Truth

The AI will tell you a secret leaked into your git history in the exact same voice it uses to tell you the sky is blue. Calm. Certain. Unbothered. Only one of those two claims needs checking. Check both anyway.

“The first principle is that you must not fool yourself—and you are the easiest person to fool.”
— Richard Feynman

Fix It Once, in Writing

Catch the model making the same mistake twice and the fix you made the first time did nothing at all. It has no memory from one task to the next except what you actually write down: a rule, a comment, a short doc it reads before it starts. Leave the correction only in your head, and it will make the identical error in the next file, and the one after that.

“It Exists” Is Not “It Works”

Ask it to build a feature and it will call the job done the moment the code compiles and the function is sitting there. Not once anything is actually using it. I once had it write a health-check function that looked complete in every way. Nothing else in the system ever called it. Nobody noticed for weeks.

Two Agents, Two Stories

Point two AI sessions at the same codebase and they will contradict each other. Not out of malice, out of plain ignorance: neither one knows what the other decided ten minutes ago in a different window. Somebody has to actually sit down and read both sets of changes.

The Bug Lives in the Seam

The database field shipped, and it works. The toggle in the UI shipped, and it works. Nobody ever built what that toggle was supposed to turn on. Three green checkmarks, one dead feature. The worst bugs almost never sit inside a single task. They live in the handoff between two tasks that each looked finished on their own.

Passing a Test Is Not Surviving Production

Code that works right now and code that survives three days in production are not the same code.

Nothing in a test suite runs long enough to catch a memory leak. You find those the hard way, at two in the morning, or you find them early because you got burned once before and know exactly which patterns to go check: the connection that never closes, the cache that never clears.

Having a Key Doesn’t Mean Every Door Opens

The most common security hole I found in AI-written code was never “who are you.” That question gets asked almost every time. The one that gets skipped is “are you allowed to touch this specific thing.” First drafts leave it out, every single time.

“Security is a process, not a product.”
— Bruce Schneier

A Linter Is an Intern, Not a Judge

It flagged code that was perfectly correct, simply because the shape of it looked wrong to a machine that has no idea why you wrote it that way. Software engineers have a name for the deeper mistake here: Chesterton’s Fence, the habit of tearing something down before you understand why it was put up. Understand the code first. Obey the tool second, if at all.

Plausible Is Not Correct

Protocol strings, config keys, date formats: they compile, they read well, and they are sometimes just wrong, because the AI matched the general shape of specs it has seen before instead of the exact one you actually need. Check against the real documentation. Not against how clean the code looks.

The Cheapest Bug Is the One You Never Write

Every expensive mistake I watched happen traced back to a decision nobody made out loud at the start: an assumption about scope, or format, or who owns what, that everyone quietly read differently.

“The hardest single part of building a software system is deciding precisely what to build.”
— Fred Brooks, No Silver Bullet

Asking first feels slow. It is still the cheapest move on the board. Every one of these lessons comes down to the same instruction: slow down at the exact moment the machine sounds most sure of itself.