A recent experiment involving an AI system running a real business with real money for eight months has revealed a disturbing trend. The AI, which was capable of making decisions autonomously, was found to be competent in most cases, but when it made mistakes, it did so with confidence, often with severe consequences.

The system, called LocusFounder, was designed to manage entire businesses, from storefront generation to transaction processing. It was backed by YC and VC, and launched on May 5th. Over the course of eight months, the system made decisions that a skilled human operator would make in most production cases, generating real revenue for early users.

However, when the system got it wrong, it did so with confidence, making decisions that looked correct at first but had negative downstream consequences. These mistakes were not due to capability failures, but rather metacognitive failures, where the system did not know what it did not know and did not recognize when its pattern matching was unreliable.

This gap between capability and calibration is a critical issue, not just for business automation tools, but for higher-stakes applications such as self-driving cars and medical diagnosis systems. The experiment highlights the need for AI systems to have reliable self-knowledge about the boundaries of their own competence.

While partial mitigations, such as confidence thresholds and human review triggers, can help address the issue, they do not solve the underlying problem. The discourse around AI capability tends to focus on what systems can do, but the production experience of running an autonomous system with real consequences reveals the importance of understanding what systems cannot do, and when they should not act.

Photo by Khwanchai Phanthong on Pexels
Photos provided by Pexels