Photo by cottonbro studio on Pexels

The integration of artificial intelligence into the legal system is facing growing scrutiny as AI-generated errors, often referred to as “hallucinations,” increasingly infiltrate courtroom proceedings. Recent cases reveal lawyers utilizing AI models such as Google Gemini and Anthropic’s Claude to draft legal documents, resulting in submissions riddled with false citations and fabricated information. A notable instance saw a California law firm penalized with a $31,000 fine for submitting a brief containing numerous AI-generated inaccuracies. Experts caution that while many errors are detected, the potential for these inaccuracies to influence judicial decisions remains a serious concern. The situation underscores the critical need for legal professionals to meticulously verify AI-generated content, recognizing that fluency does not equate to factual accuracy. Despite these inherent risks, legal technology firms continue to promote AI tools with promises of precise and reliable output. This issue was initially highlighted in *The Algorithm*, a newsletter published by MIT Technology Review.