The first time you watch an AI coding tool turn a few paragraphs into a working application, it is easy to start doing the math. If Claude Code or GitHub Copilot can produce that much functionality that quickly, what could a company accomplish with an entire development process built around AI?
That is a reasonable question. The trouble begins when an impressive demonstration becomes the justification for replacing the people who know how to build, validate, and maintain the software.
I call this the naive optimism trap. It is the assumption that because AI can quickly handle a visible portion of software development, it can reliably take responsibility for the whole job. As companies expand from coding assistants into AI harnesses that coordinate implementation and testing, that assumption can spread across the entire project.
I have felt that optimism in my own work, and I have seen other project leaders experience it too. In my role as Practice Lead of Digital & App Innovation at OZ Digital, I have been involved in operationalizing AI harnesses, Claude Code, and GitHub Copilot within development projects. That includes work modernizing IBM mainframe applications to C# and .NET, where the distance between generating code and delivering the right system becomes especially clear.
AI offers tremendous value in this work. It also creates opportunities to underestimate how much engineering remains.
A coding demonstration usually makes progress easy to see. Screens appear, endpoints respond, and functionality takes shape while you watch. “Vibe coding” can be useful for exploring an idea or producing a prototype, especially when the cost of changing direction is low.
Production software brings a different set of obligations. Someone must establish that the behavior meets the requirements, the design fits the surrounding system, the security controls are appropriate, and another engineer can safely modify the code six months later. A successful demonstration provides limited evidence about those obligations.
Modernization makes this particularly important. Translating existing logic into C# is only part of the work; preserving the behavior the business depends on requires understanding and validation. Consider a calculation that looks cleaner after a rewrite but handles rounding differently, or a redesigned workflow that overlooks an exception users have relied on for years.
A system that compiles and passes its current tests can still require substantial investigation before anyone should trust it. In a modernization project, understanding the behavior we need to preserve is part of the engineering work.
In practice, the early gains can feel like getting 50–80% of the way there surprisingly quickly. Moving toward 90% or more of development work being handled reliably by AI takes substantially more effort. I use those percentages to describe the experience of progress, rather than as a measured autonomy benchmark or a promise about what every team can achieve.
Much of the remaining work involves context, validation, and consistency. We need the AI to produce good, clean, maintainable code, follow the project’s architectural decisions, and continue doing so across repeated changes. The familiar difficulties of software development persist regardless of who, or what, types the implementation.
This is where the AI harness becomes important. A harness is the software and workflow around the model: the instructions, tools, context, execution loop, checks, and escalation rules that guide its work. It determines what the agent can do, how it receives feedback, and when a person needs to intervene.
In my work, that has involved custom loop prompting, deterministic script-based gate checks, and custom agent and skill definitions for Claude Code and GitHub Copilot. Each piece requires engineering. A mainframe modernization project and a greenfield application also need different context and validation strategies, even when they use the same underlying AI tools.
Anthropic has documented related challenges in its engineering work on long-running agents. Its experiments found agents attempting too much at once, losing continuity between sessions, and declaring work complete without adequate testing. Structured progress records, smaller increments, and explicit end-to-end validation helped address those problems. That is useful engineering evidence for investing in the workflow surrounding a model, although it does not establish that any particular harness is ready to replace a development team.
Model improvements still matter. However, in the projects I work on, some of the most actionable gains have come from improving the harness: giving the AI better context, tightening its feedback loop, and making project expectations executable. That is an observation about implementation experience, rather than evidence that model progress across the industry has reached a ceiling.
A stronger model does not automatically know an undocumented business rule or an architectural constraint nobody supplied. Teams can improve those inputs today, without waiting for the next model release.
Deterministic checks are especially valuable because they turn some engineering expectations into repeatable validation. Rather than simply asking an agent whether it followed the rules, we can run a build, execute tests, apply static analysis, or check for prohibited dependencies. The harness can return specific failures to the agent and require another attempt.
For example, a .NET team might enforce a rule that its domain project cannot reference its infrastructure project. An automated dependency check can catch that violation consistently. A reviewed regression test can likewise verify a known business calculation against an expected result.
These checks encode parts of human expertise, but they have limits. A dependency rule cannot decide whether the domain model makes sense, and a passing test cannot establish that its expected result was correct. The checks themselves need review, and an agent should not be free to weaken the criteria merely to make its work pass.
The same caution applies when AI writes both the implementation and the tests. Both can reflect the same misunderstanding. Independent acceptance criteria, established behavior, and human validation remain essential.
Published research reinforces the need to distinguish impressive output from demonstrated results. A study combining three field experiments across 4,867 developers found a roughly 26% increase in completed tasks with access to an AI coding assistant. That is meaningful evidence of productivity gains, although the study examined code-completion assistance rather than autonomous replacement of software teams.
METR’s randomized study of experienced open-source developers using early-2025 tools produced a different result: participants took 19% longer with AI, while believing it had made them faster. In a February 2026 update, METR reported that newer results were complicated by selection effects and suggested that tools likely provided greater speedups than before, without reliably establishing their size.
These studies involve different tools, tasks, and conditions. They support a practical conclusion: measure results in your own environment. Neither a productivity gain nor a slowdown in one setting answers whether your project can safely operate with fewer experienced people.
That distinction matters because experienced engineers contribute judgment throughout delivery. They clarify requirements, evaluate design tradeoffs, recognize architectural drift, and review whether generated code belongs in the system. They also know when a seemingly straightforward request needs a conversation before anyone writes more code.
A useful way to manage an “AI developer” is to apply the supervision you would give a junior developer. Provide clear requirements, explain the project’s conventions, assign bounded work, and review the result. Trust but verify.
The analogy has limits. AI may produce sophisticated code at extraordinary speed while missing a requirement that a person would question immediately. Its apparent fluency should never determine its authority to approve architecture, accept security risk, or declare a business requirement satisfied.
The lesson extends to business analysts and QA professionals. AI can help draft acceptance criteria, generate test cases, investigate failures, and automate repetitive validation. People still need to determine whether the requirements reflect the business need and whether the tests challenge the right assumptions.
Human review also needs to be substantive. A reviewer who sees a passing pipeline and clicks Approve without understanding the change provides little protection. Teams should make review manageable through focused changes, traceable requirements, useful test evidence, and enough time for engineers to examine the decisions that matter.
This is the approach I recommend for adoption: integrate the harness alongside your human team, then expand its responsibilities as the process demonstrates reliability. Begin with bounded tasks, establish review and validation practices, and use failures to improve the instructions, checks, and context.
Measure the complete path to accepted software. Include review time, rework, defects, and maintenance effort alongside implementation speed. Generating code faster creates value when the team can turn it into dependable software more efficiently.
The Software Dark Factory remains a compelling direction for the industry. Increasingly automated delivery can remove substantial repetitive work and allow people to focus on requirements, design, exceptions, and improvement. Reaching that level of automation takes deliberate engineering of the delivery system itself.
I remain optimistic because I have seen how much AI can enhance a capable developer. It can accelerate initial implementation, help modify existing code, and assist with locating and fixing bugs. Guided by an experienced person, it becomes a powerful addition to the team.
The danger is allowing that early success to justify removing the expertise needed to sustain it. Bring AI into the project, give it useful work, and build the evidence for greater autonomy. Keep the people who understand what the software must do responsible for deciding when it is ready.
