Photo by Artem Podrez on Pexels

A new analysis of 445 Large Language Model (LLM) benchmarks, presented at leading AI conferences, raises concerns about their construct validity – the extent to which they truly measure the theoretical concepts they aim to assess. The study, brought to wider attention by a Reddit user on /r/artificial/, points to issues such as unclear definitions of target phenomena and a deficiency in rigorous statistical testing. These shortcomings suggest that a significant portion of current LLM benchmarks may not be reliably evaluating the specific abilities they are intended to gauge. The research paper detailing these findings is available at https://oxrml.com/measuring-what-matters/.