Over the past year, a particular conversation has become common enough that I can now predict how it goes.
A software company has built something quickly — a customer portal, an internal tool, a product feature — largely with an AI coding assistant. It works. The demo went well. Someone has asked whether it can go in front of real customers, and the team's honest answer is that they don't know. Not “we think there are problems.” They genuinely cannot tell.
That last part is what interests me. These are capable teams. They aren't asking us how to build software. They're asking a narrower and much harder question: what specifically is wrong with the thing we already have?
The risk isn't that the code is bad
There's a common assumption that the danger with AI-generated code is quality — that the model writes sloppy code and someone needs to clean it up. In our experience that's rarely the problem. The code we see is usually clean, idiomatic, and does exactly what was asked.
The risk is what isn't there.
An assistant produces code that satisfies the request in front of it. Ask for an endpoint that returns a customer record and you get one — complete, correct, and finished with respect to the question asked. What's missing is everything nobody thought to request, because they didn't know it was part of the job. A senior engineer's real contribution was never typing speed; it was knowing which unasked questions matter. That knowledge used to be applied automatically, as the code was written. Now the code arrives without it, and it arrives fast enough that the gap isn't visible until real traffic finds it.
You cannot find absent code by reading the code that's there. Everything looks finished, because it is finished — just not complete.
Two answers, both the wrong size
When a team asks us what's wrong with their application, there have historically been two things we could offer, and neither fits the question well.
The first is automated scanning. Point a tool at the repository and it returns findings — often hundreds of them. The trouble is that a scanner has no view of the business. It cannot tell you whether a flagged dependency sits in a code path you actually execute, whether a permissive setting is dangerous given how you've deployed, or which three of two hundred findings would genuinely hurt you next month. Volume without judgment doesn't reduce uncertainty; it relocates it.
The second is a proper engagement — our engineers spend time with the application and tell you what they think. That works. It's also the wrong size for the question. A company asking “is this safe to launch?” is trying to make a decision, not commission a project, and a full assessment costs more than the decision is usually worth at that stage.
So teams did what people do when both options are wrong: they waited, or they launched and hoped. Neither is a good outcome for a business, and both are avoidable.
What we kept finding
Across the applications we look at, the same handful of gaps recur with striking consistency. Not obscure ones — the same five, over and over:
- Credentials committed to the repository. Usually an API key added early to get something working, later hidden from view but never removed from the project's history.
- Users able to see one another's data. The application checks that someone is signed in, but not that the record they requested belongs to them. In most jurisdictions this is a reportable breach rather than a defect.
- No spending limits on AI model calls. Nothing fails and no alert fires. The consequence arrives as an invoice.
- User input passed directly into model instructions. Which allows a customer, deliberately or otherwise, to make the product say things nobody approved.
- No error monitoring. The first indication that something is broken is a customer telling you — after every other affected customer has already experienced it.
None of these are exotic. All are cheap to fix once identified. The expensive part has never been the repair; it's the not knowing.
So we built the diagnosis
Earlier this year my colleague Erol wrote about hardening vibe-coded applications for production — the lifecycle, the controls, the governance that makes an AI-built system safe to depend on. It remains the piece we send most often.
It answers what to do. It doesn't answer what's actually wrong with your application, and that turns out to be the step teams get stuck on. “Harden the app” isn't a task; it's a category, and it has no obvious first move.
That gap is what we've productized. VibeXray runs an automated scan across 28 production-readiness checks — security, AI-specific risk, reliability, operability, cost — and then a senior engineer reads the results, removes the false positives, and writes the priority list: what's urgent, what's benign, and what it costs you if you launch on Tuesday.
The model is deliberately borrowed from medicine. The machine takes the X-ray; a radiologist reads it. Imaging alone doesn't help a patient, and no clinic asks a specialist to run the scanner. The value is in the interpretation — and interpretation is the part that has never scaled, which is exactly why most teams have never had access to it.
It's priced as a decision rather than an engagement: a fixed fee, a 48-hour turnaround, and a report the team can act on themselves. If the findings warrant a remediation sprint, we can do that too — but the report is written to be useful whether or not anyone hires us, which is the only version we were interested in building. You can read a complete sample report before deciding whether it's for you.
Where this fits
We've written before about the path we see software companies travelling: migrate, build the foundation, modernize software delivery, enable AI. And about the stages of AI adoption itself — from AI-assisted work, where a person directs each step, toward AI-directed and eventually AI-delegated, where supervised agents complete defined work and return it for approval.
Everything about that progression depends on one capability: the ability to look at work you didn't write line by line and judge whether it's sound. That's true of a Copilot suggestion and it's true of an entire application. Teams that can do it move up the ladder. Teams that can't either stop, or advance on faith — and faith is not a control.
Giving software companies a fast, honest, affordable way to answer “is this ready?” is a small piece of that larger picture. But it's the piece that was missing, and it's the one we were asked for most.
If you've built something with AI and you're not certain what's underneath it, that uncertainty is worth resolving before your customers resolve it for you.
Deop helps software companies establish governed AI foundations, modernize their GitHub and DevOps workflows, and move from AI experimentation to secure production delivery — as a Microsoft Solutions Partner in Cloud & AI Platforms and Security. Explore our AI and Copilot enablement work, or see a sample VibeXray readiness report.