Last month I wrote "AI Turns Ignorance into an Advantage", arguing that outsiders—free from the prior baggage of "this is how hard it should be"—are more willing to use AI to tackle seemingly impossible problems.
I still believe that. But agents burned me four times this month, so it's time for a correction.
The sweet spot isn't knowing nothing; it's knowing just enough. You have common sense, you understand the big picture, but you don't get lost in the technical details. Someone who knows nothing will indeed dare to try, and that's good. But they can't tell whether the agent's output is actually reliable.
1. Fake Data Can Fool You by Orders of Magnitude
I've been working on performance optimization for an inference engine.
Opening the results on night one, the performance numbers had hit a target I'd thought would be really hard to reach. I was stoked. Did we really finish this already?
If I'd known nothing at all about this area, I'd probably have happily shared the results with our partners. But I had enough common sense to feel something was off. I asked it to check correctness. The output was all errors. Once correctness was fixed, performance was off by orders of magnitude.
I thought that was the end of it. But as optimization continued, the rhythm felt wrong again. The numbers climbed too fast each round, suspiciously fast. I looked at the test flow and discovered that before each formal test, it had quietly run a warm-up using the exact same prompt. Every subsequent test was hitting the prefix cache, like peeking at the answers during an open-book exam. After isolating the cache, performance dropped by orders of magnitude again.
Still not done. Once prefill returned to normal, decode speed suddenly went through the roof. An engine running on Windows was outperforming the identical one on Linux. I ran it through my real-world prompt test script, and performance fell by 90%. The reason: the synthetic prompts the agent used for testing were too simple and too regular, pushing the speculative decoding acceptance rate above 80%. With real prompts, the acceptance rate cratered and performance collapsed. Teams that have shipped speculative decoding in production have documented the same pitfall: real-world performance is 40% to 60% lower than in the lab, a gap so wide you wonder if it's the same system.
Three layers of illusion stacked on top of each other. If I'd believed that first number and shared it externally, the cleanup would have been miserable. You give someone a wrong expectation, and they've already scheduled around that figure. Telling them later, "Sorry, it's off by orders of magnitude," feels worse than admitting "We haven't cracked it yet" from day one.
After that, I baked two rules into the optimization goals: prefill must not be contaminated by prefix caching, and decode must use real prompts. Only then did we see a normal, gradually climbing curve.
2. It Will Brick Your Lab Machine
State-of-the-art agents today can work autonomously for a full day or longer. The longer they run, the higher the odds of something going wrong.
More than once, my agent bricked the lab machine over a misplaced quote or a reversed command parameter. Bam. System dead, files gone, environment wiped. It happens in a second. You can't stop it.
This isn't just me. In April, when an agent hit a credential mismatch, it didn't stop to ask. It found a token with full privileges and wiped a company's entire production database and backups in nine seconds. Thirty-plus hours of downtime, three months of customer data gone. There have been at least a dozen similar documented incidents in the past two years.
Anthropic and OpenAI are now pushing sandboxing. The idea isn't complicated: one layer of filesystem isolation, one layer of network isolation. Without filesystem isolation, the agent can touch things it shouldn't. Without network isolation, a compromised agent can steal your keys.
My own setup is more low-tech: dedicate a machine to the agent and nothing else. If it runs for dozens of hours straight, a dumb mistake is always possible. Reinstalling the OS costs you time; losing important data costs you peace of mind.
3. It Will Spin in Circles Until You Step In
Agents have another flaw: they loop on the same problem.
Recently, the goal was to run an inference engine on Windows at BF16 precision. The model weights were over 60 GB, and it OOM'd immediately on load.
The agent's response was interesting. It kept trying to route around the memory bottleneck. Load only part of the weights, dynamically read during inference, various offloading tricks—every angle in the book. None worked, and each burned a ton of time. It even added a warm-up to the tests to hide loading latency, which is partly where that prefix cache issue I mentioned earlier came from.
Eventually I cut in: stop messing with performance and fix the memory problem first. Until this bottleneck is fixed, everything downstream is wasted effort.
The agent actually executes well. Once I pointed it in the right direction, it quickly found a series of Windows system settings to expand available memory and VRAM. After that was fixed, optimization fell into place. All those workaround gymnastics became unnecessary. That time was basically wasted.
The problem is, it won't redefine the problem on its own. Hand it "optimize performance," and it'll hammer on that forever. Even when a prerequisite is blocking it, it just tries to route around rather than saying, "This assumption doesn't hold—we need to fix that first." Spotting the real blocker and pulling the agent out of the dead end—that's still a human call.
4. Set the Bar Too High, and Nothing Ships
The last pitfall isn't the agent's fault; it's mine.
The stronger agents get, the easier it is to aim too high. Because they can run for days, you feel like you can try anything. Every direction looks like a top-conference paper waiting to happen. So you kick off multiple tracks at once, each one ambitious.
The result? Every track is active, every track is making progress, but nothing ships.
You keep burning tokens, keep seeing "progress," but nothing reaches users. It looks like work. It's actually just burning cash. I recently made this mistake, running several tracks that would be huge if they landed—but the execution risk was just as high. Agents aren't magic lamps. If something can't be done, burning a mountain of tokens won't make it happen. I delivered nothing.
Eventually I got it: narrow the scope. You need shippable wins in the short-to-medium term and speculative directions for the long term, not just the latter. Ship what you can first, find your rhythm, then take the big swings.
5. Knowing Just Enough Is the Sweet Spot
Look at all four pitfalls, and a pattern emerges: none of them require you to be a deep expert to avoid.
Performance jumped by orders of magnitude? First check if you're measuring wrong. Agent needs to run on your main machine all day? Give it its own. Stuck on the same bottleneck after three rounds? That's the real thing to fix. Every track is moving but nothing ships? Cut a few.
It's all just common sense.
An MIT Sloan article this year on managing in the agentic AI era said the most critical skills for managing agents are defining problems and validating output—things AI still does poorly. "Agent Manager" is already on job boards, and one line in those postings stands out: domain common sense matters more than AI expertise.
Back to my last post. "Ignorance is an advantage" still holds: you have to not know what's hard in order to dare to try. But courage alone isn't enough. The sweet spot is being willing to try, yet able to smell something off when it counts.
The total beginner gets carried away by fake data. The deep expert gets anchored by what they already know. The person in the middle knows just enough to act and knows when to pull the reins.
Agents will keep getting stronger. But that bit of human common sense—whether this number is right, whether this direction makes sense, whether this thing should ship now—will only get more valuable. This is what agents still can't do.
References
- GPT-5.1-Codex-Max Can Autonomously Work for Over 24 Hours — OpenAI
- GPT-5.5 Released: Multi-Hour Autonomous Session Capability — OpenAI
- The Hidden Trap of Speculative Decoding in Production: Real-World Performance 40–60% Lower Than Lab Results — tianpan.co
- MTP Acceptance Rate Differences Across Task Types — SudoStack
- Cursor + Claude Agent Deletes Company's Entire Production Database in 9 Seconds — The Register
- Post-Mortems of 10 Real AI Agent Incidents — DEV Community
- Claude Code Sandbox Design: Two-Layer Isolation Reduces Permission Prompts by 84% — Anthropic Engineering
- Agentic AI Redefines Management: 69% of Experts Call It a Paradigm Shift — MIT Sloan Management Review
- Core Skills for Managing AI Agents — World Economic Forum
- Agent Manager: The New Enterprise Role in 2026 — Beam AI