When we talk to people about how we build, the question we hear most often is some version of: why bother? If the AI can handle the task, why add friction? Why require a human to review? Why build in the ability to undo something that was done correctly?
It is a fair question. And the honest answer is not purely philosophical. It is practical.
Trust is earned, not assumed
The systems we build go into environments where people have learned, often through hard experience, not to trust new software. They have seen systems that promised to make their jobs easier and made them harder. They have been asked to adopt tools that were designed for an office context and shoehorned into a care context. They are right to be sceptical.
The human-in-the-loop design is partly how we earn that trust. When a care worker can see exactly what the system has done and can reverse any action with a single step, they are in control. The system is doing work for them, not to them. That distinction matters enormously in environments where the consequences of errors are significant.
A system that staff do not trust does not get used. A system that does not get used does not reduce anyone's workload, does not improve any outcomes, and costs the organisation money for nothing. The override button is not friction. It is the reason the system gets adopted.
AI systems make mistakes
This should not need saying, but it does. AI systems, including the ones we build on and the ones we build, make mistakes. They misread context. They apply patterns that fit 95% of cases to the 5% where they do not apply. They are confident in situations where confidence is not warranted.
In a low-stakes context, a mistake from an AI system is an inconvenience. In a care environment, a mistake in a medication record or a safeguarding report is something else entirely. The human checkpoint is not a vote of no confidence in the AI. It is an acknowledgement that no automated system, however capable, should be the last line of defence in a high-stakes process.
The organisations that get this right treat the AI as a very capable colleague who drafts things for review, not as an authority who makes final decisions. That framing is accurate to what the technology can and should do.
Auditability is not a constraint, it is a feature
Every action a Moai system takes is logged. The log is readable by the people using the system, not just by IT departments running database queries. It shows what happened, when, and what the system's basis for action was.
We built this because it is required in regulated industries. But we kept it and expanded it because we discovered it was valuable independent of regulation. Managers use the log to understand how the system is performing. Staff use it to catch their own errors. In the rare cases where something goes wrong, the log is the first thing everyone wants to see.
An opaque system that works is always one system failure away from being an opaque system that failed in ways nobody can explain. Transparency is not overhead. It is structural resilience.
The argument from reversibility
We design every automated action in our systems to be reversible. This is a harder constraint than it sounds. It means thinking carefully about what a reversal means in every context. Reversing a drafted document is straightforward. Reversing a submission is more complex. Reversing a communication requires a different mechanism entirely.
The effort this takes at the design stage is significant. But the alternative, building a system where automated actions are irreversible, creates a category of risk that we are not willing to introduce into environments where people's welfare is involved. If the system makes an error that cannot be undone, the cost of that error falls on the people who trusted the system. That is not acceptable.
What this means in practice
Building with humans in the loop does not mean building slow systems. Our users review and approve actions in seconds, not minutes. The audit logs do not slow anything down. The override mechanisms are present but rarely used because the systems perform well enough that they are not needed often.
What it does mean is that the systems we build have a higher design cost and a higher development cost than systems without these features. We accept that. We think the organisations using our systems should expect it and ask for it from anyone selling them automation for regulated or high-stakes contexts.
If you are evaluating automation and want to understand what to ask about human oversight, reversibility, and audit trails, get in touch. We are happy to share what we look for.