ChatGPT Hallucinations Put New Mexico Murder Appeal Under Scrutiny

ChatGPT hallucinations are becoming a serious concern as generative AI is increasingly used in professional fields where accuracy matters. A recent New Mexico murder appeal has brought that risk into focus after an attorney submitted a court filing containing fabricated witness and police testimony generated with ChatGPT.

The New Mexico Supreme Court determined that the filing included information that did not exist in the underlying case. The attorney, Stephen Aarons, was fined $5,000, held in contempt, and referred to the state’s attorney disciplinary board for further investigation.

The case is another example of a problem that has become increasingly important as generative AI is adopted in professional environments: AI systems can produce convincing information that is completely false.

AI-Generated Testimony Appeared in a Murder Appeal

According to the New Mexico Supreme Court, Aarons submitted a brief containing what the court described as false testimony attributed to completely fabricated witnesses.

Aarons had been handling the appeal of Oscar Renee Sandoval, who was convicted of murdering his children and sentenced to life in prison. Sandoval’s appeal remains pending and was assigned to New Mexico public defender Kim Chavez Cook on September 2.

The court criticized Aarons for failing to verify the information included in his filing and said his conduct demonstrated a lack of concern for the consequences to his client.

The attorney acknowledged that he had used ChatGPT while preparing the appeal. He told Reuters that he initially used the AI system to summarize trial proceedings after agreeing to take on the case.

Aarons said he did not understand the extent to which AI could “hallucinate” information.

He described the incident as an honest mistake and said it demonstrated an important lesson for professionals using the technology.

What Are ChatGPT Hallucinations?

An AI hallucination occurs when a generative AI system produces information that appears credible but is inaccurate, unsupported, or entirely fabricated.

The problem can be particularly difficult to detect because AI-generated text often sounds confident and authoritative.

A language model does not automatically know whether every statement it produces is factually correct. It generates responses based on patterns learned during training and the information available to it during a particular interaction.

As a result, an AI system can sometimes produce a realistic-looking name, quotation, legal citation, study, event, or other detail that has no basis in reality.

That becomes especially dangerous when people assume that fluent writing is evidence of accuracy.

Why the Legal Profession Faces a Serious AI Risk

Legal work depends heavily on accurate information.

Attorneys routinely deal with statutes, court decisions, evidence, testimony, legal precedents, and detailed factual records. A fabricated citation or inaccurate description of a case can undermine an argument. Fabricated testimony introduces an even more serious problem because it can create an entirely false account of what happened.

The New Mexico case demonstrates how an AI hallucination can move beyond an incorrect answer in a chatbot conversation and become part of an official legal proceeding.

That distinction matters.

An incorrect AI-generated summary that is caught during a private drafting process may be relatively easy to correct. An unsupported claim submitted to a court can have consequences for the attorney, the client, and the judicial process.

AI Has Already Caused Problems in Court Filings

The New Mexico case is not an isolated example of problems involving generative AI and legal documents.

Lawyers in multiple jurisdictions have faced sanctions after using AI systems that generated nonexistent court decisions, incorrect citations, or inaccurate legal information.

These incidents have become a warning for the legal profession as AI tools become easier to access.

The underlying problem is not necessarily that AI should never be used for legal work. AI can potentially help attorneys with tasks such as organizing documents, summarizing information, reviewing large amounts of text, and assisting with research.

The problem arises when AI-generated material is treated as reliable without independent verification.

Why AI Can Sound Convincing When It Is Wrong

One of the most challenging characteristics of generative AI is that incorrect information does not necessarily look incorrect.

A traditional search engine generally points users toward sources that they can inspect. A generative AI system can instead produce a complete answer in natural language.

That answer may contain names, dates, quotations, and explanations presented with the same confidence whether they are accurate or fabricated.

This can create a dangerous psychological effect: people may be more willing to trust information because it is presented clearly and professionally.

For professionals working in fields where accuracy is critical, that means AI-generated information should be treated as material requiring verification rather than as automatically reliable evidence.

The Human Verification Problem

The responsibility for a professional document ultimately remains with the person submitting it.

In a legal setting, that means attorneys need to independently verify factual claims, sources, citations, quotations, and other information before including AI-generated material in a court filing.

The same principle applies outside the legal profession.

Doctors, financial professionals, journalists, researchers, engineers, and business leaders may all use AI to accelerate their work. But the higher the consequences of an error, the more important human review becomes.

AI can make the process of producing information faster. It does not remove the responsibility to determine whether that information is correct.

AI Is Powerful, but It Is Not a Source of Truth

The New Mexico case illustrates a broader misunderstanding about generative AI.

ChatGPT and similar systems are extremely capable of producing useful text, analyzing information, and assisting with complex tasks. But their ability to produce convincing language should not be confused with the ability to guarantee factual accuracy.

This is particularly important when AI is used for high-stakes decisions.

An AI-generated answer can be useful as a starting point. It can help a professional identify questions, organize information, or explore possible explanations. But important claims still need to be checked against reliable primary sources and the underlying evidence.

That distinction becomes even more important as AI moves beyond chatbots and into systems capable of taking actions and performing more complex professional workflows.

The Bigger Lesson for AI Users

The case involving Aarons highlights one of the biggest challenges facing generative AI adoption: the technology can be useful and unreliable at the same time.

AI systems can save professionals significant amounts of time, but that benefit comes with a responsibility to understand their limitations.

The more consequential the task, the less acceptable it is to simply copy an AI-generated answer without checking it.

For legal professionals, that may mean verifying every citation and factual claim before submitting a document. For businesses, it may mean requiring human approval before AI-generated information is used in important decisions.

The lesson is straightforward: AI should assist professional judgment, not replace it.

A Warning for the Next Stage of AI

As generative AI becomes increasingly integrated into professional software, incidents like this could become more important rather than less.

Future AI systems may be capable of researching documents, drafting legal arguments, analyzing evidence, and interacting with professional databases with far less human involvement. Those capabilities could provide enormous benefits, but they also increase the consequences when an AI system makes a mistake.

The challenge will therefore not simply be making AI more capable.

It will also be making sure that people understand when AI should be trusted, when it should be checked, and when it should not be allowed to make decisions on its own.

The New Mexico case provides a stark example of why that distinction matters. A polished AI-generated sentence can look authoritative, but in a courtroom, credibility ultimately depends on evidence—not how convincing the language sounds.