For decades, the performance of artificial intelligence has been measured by its ability to outperform humans in various tasks, from chess and math to coding and essay writing. However, this approach has a significant flaw: AI is rarely used in the same way it is benchmarked.

Current benchmarking methods evaluate AI’s performance in isolation, outside of the complex environments where it is actually used. This misalignment leads to a misunderstanding of AI’s capabilities, overlooks systemic risks, and misjudges its economic and social consequences.

To address this issue, a new approach is needed, one that assesses how AI systems perform over longer time horizons within human teams, workflows, and organizations. This is where HAIC benchmarks — Human–AI, Context-Specific Evaluation — come in, providing a more comprehensive understanding of AI’s performance in real-world settings.

Traditional benchmarks can be misleading, as they do not account for the complexities of real-world deployment. For instance, an AI model may achieve impressive technical scores, but its performance in a actual setting may be hindered by factors such as hospital-specific reporting standards and regulatory requirements.

It is time to shift from narrow, task-level evaluations to more holistic approaches that consider the broader context in which AI is used. By doing so, we can gain a better understanding of AI’s true capabilities and limitations, and make more informed decisions about its deployment and use.

Photo by Ebahir on Pexels
Photos provided by Pexels