AI agents can turn experiments into software that others depend on. As that dependency grows, so do the responsibilities. How should teams expand agent autonomy while keeping systems understandable, maintainable, and supported? Start with the consequences of failure and the evidence that each workflow can be trusted.
Imagine using an AI agent to build a utility that converts CSV files. It starts with disposable test data. Someone finds it useful, shares it, and eventually schedules it to update live customer records.
At some point, deleting the project stops being a harmless decision. Other people now depend on it.
That transition interests me more than how quickly the first version was built. What needs to change when an experiment becomes part of someone else’s workflow?
I want agents to make it easier for people to solve problems, including people who would never have built software before. I also want a clear way for those solutions to acquire the engineering support their growing responsibilities require.
I would decide how much freedom an agent gets by examining the work it can perform and the consequences of getting it wrong.
Four questions help:
What systems, people, and data could this action affect?
How much of the damage could we reverse?
How reliably can we detect an incorrect result?
Which decisions still require judgment about unclear requirements or competing goals?
An isolated experiment with disposable data can allow considerable freedom. An agent could try several implementations, discard them, and start again. Extensive review at every step might add little value.
Changing access rules or updating live records calls for stronger evidence and tighter permissions. A small patch can have large consequences.
These distinctions also apply within one project. An agent might implement changes and run tests independently, while a human approves a migration or a new external integration. Automatic merging could be reasonable for a narrow class of changes with reliable checks and limited consequences.
Permission to write code, merge it, and release it can be separate decisions.
Even a workflow that automates all three still needs people to define its purpose, constraints, and acceptance criteria. The team operating it remains responsible for deciding whether those controls are adequate.
Suppose agents became consistently excellent programmers. I would still want the people responsible for a service to understand its design.
They need to know which assumptions matter, where its boundaries are, and how to investigate unexpected behavior. They also need to explain its limitations to the people who rely on it.
That understanding can be distributed across a team and supported by documentation, tests, and operational evidence. It needs to stay current as the system changes.
A review process should help maintain that knowledge. If people approve changes they cannot explain, the presence of a human approval step provides limited reassurance.
Architecture matters for the same reason. An implementation can meet today’s requirement while increasing the effort needed for the next change. A rule copied into several modules might pass every current test. Later, someone must discover all those copies and keep them consistent.
I would evaluate generated changes partly by what they leave behind: clear boundaries, consistent behavior, and decisions that future maintainers can understand.
The workflow also needs to respect the team’s capacity to absorb changes. In Loop Engineering from First Principles, Kyle Mistele describes pausing a maintenance loop while its previous pull request remains open. That is a useful example of making review capacity part of the automation.
Machine learning offers a useful parallel. Google’s MLOps guidance describes moving from manual experimentation and deployment handoffs toward repeatable pipelines with validation, metadata, and monitoring.
I think agentic development could benefit from a similar approach.
A useful prototype should have a clear route into supported operation. Its creator should be able to discover what is required, get help, and reuse established deployment and monitoring practices.
Ownership should be clear from the start, even if the initial responsibility is simply to keep the experiment contained. As its use expands, the owner needs to reassess its data access, dependencies, failure behavior, and support needs.
This review should happen when those conditions change. A project can acquire real obligations while everyone still calls it an experiment.
I would keep the initial requirements small. Many ideas will be abandoned, and that is a useful outcome of experimentation. Additional controls become justified as the software gains users, permissions, or dependencies that would be costly to disrupt.
A platform can help make those controls easier to adopt. So can clear expectations about who supports the result and who can authorize its next stage of use.
I would like to see highly autonomous development tested on bounded, useful work. Give an agent workflow clear objectives, constrain its permissions, and measure what happens.
The evaluation should include more than the time taken to produce an initial implementation:
How much human effort goes into specifying, reviewing, correcting, and supporting the work?
Which defects escape validation, and how quickly can the team recover?
How difficult are later changes?
Can another maintainer explain the design and investigate a failure?
These measures should be compared with the team’s existing workflow on similar work. A fast first version could still create expensive maintenance. A workflow that requires more setup could repay that cost through reliable repeated use.
I would also want evidence from teams that have operated the results over time. Demonstrations and claims from tool builders can suggest what to try. Sustained use helps reveal which assumptions hold.
The policy should be able to change in both directions. Reliable results can justify broader autonomy. Weak validation, unexpected behavior, or growing consequences can justify narrowing it.
I am optimistic about how much software agents will help people build. I want the process around them to improve at the same pace. More capable agents should let us delegate more work while retaining the knowledge and ownership needed to live with the result.
Your cart is currently empty.
Start Shopping
