Courtroom 'hallucinations': Lawyers citing precedents dreamed up by AI and clients sabotaging their own cases
US federal judges warn against blindly relying on AI; Israel’s High Court rebukes lawyer who relied on AI “hallucinations”; NY federal court rules conversations with chatbots are not privileged
A troubling phenomenon has become increasingly common in courtrooms in recent years: Lawyers, some with decades of experience, are being reprimanded by judges, fined and even barred from appearing in court, all because of laziness and technological shortcuts. Judges are discovering that complex legal arguments submitted to them rely on entirely fictitious rulings and precedents generated by machines.
One recent case making waves in the US legal community comes from Mississippi. In *Withers v. City of Aberdeen*, attorneys for both sides cited no fewer than six entirely fabricated precedents.
Federal Judge Sharion Aycock removed the lawyers from the case, imposed financial penalties and revoked their pro hac vice status, a temporary privilege allowing an out-of-state attorney to appear before the court.
“At this point, no lawyer can credibly claim to be unaware of the risks,” a University of Mississippi law professor who serves on the state’s ethics committee told local media.
Israeli lawyer rebuked by Supreme Court: ‘Betrayed the profession’
The Israeli justice system has not been spared the phenomenon. In February 2025, an extraordinary case emerged at the Supreme Court after an attorney representing a petitioner in a Sharia divorce dispute blindly relied on an artificial intelligence tool and directed the court to rulings that had never existed.
Supreme Court Justice Gila Canfy-Steinitz dismissed the petition and sharply criticized the attorney in her ruling.
“A lawyer who files a pleading containing fabricated material, while relying on authorities that do not exist, betrays their duties to the litigant, to the court and to the legal profession,” she wrote.
The justice stressed that using AI does not absolve attorneys of professional responsibility or their duty to thoroughly verify their submissions. She nevertheless refrained from imposing financial penalties, given that it was the first case of its kind to come before the Supreme Court.
Decisions without consciousness: Warnings from Britain
The professional danger extends into the foundations of legal theory and practice. British legal experts Dr. Eden Sarid and Dr. Haim Abraham told i24NEWS that the technology could dramatically alter the relationship between citizens and the law.
“The use of artificial intelligence is creating a profound change,” Sarid said. “In administrative law, a person is entitled to receive an explanation for a decision concerning them. But artificial intelligence is a tool without self-awareness. When judges rely on it blindly without understanding how it works, we retreat from the process of rationalization that has characterized law since the Enlightenment and return to relying on a kind of digital fortune teller.”
Abraham said the threat was already being felt in practice, recounting his own experience with a lawyer whom he and his colleagues had hired.
“We asked complex questions and received a flat, hollow output from ChatGPT,” he said. “After we confronted her, the firm admitted using it. It was a serious breach of privacy and professionalism, and we fired them immediately. At least in the legal field, this technology is not there yet, and using it is extremely dangerous.”
The New York bombshell: The chatbot that could destroy your case
The damage caused by artificial intelligence is not confined to lawyers. It can also affect clients and potentially destroy their defense. A precedent-setting ruling by a federal court in New York has exposed a new danger.
A criminal suspect consulted the Claude artificial intelligence tool, opening 31 separate chats and entering sensitive details about his case, including information he had received from his attorneys. The FBI later raided his home and seized printouts of the conversations.
The New York judge ruled unequivocally that the information was not privileged.
The ruling rested on the fact that artificial intelligence is not a lawyer and that no attorney-client relationship, and therefore no legal privilege, exists. Entering information into a chatbot is legally comparable to discussing it with a stranger on the street.
The platforms’ terms of service also make clear that they do not provide legal advice and that conversations are not confidential and may be used to train their models.
The implication is that clients who ask AI how much their lawsuit might be worth, or disclose potentially damaging facts, could later discover that their chats have become powerful weapons during cross-examination.
The illusion of the 'robot lawyer'
Lawyers have warned for years against relying on chatbots in place of qualified legal counsel. Numerous startups promising to “replace the legal system” have encountered regulatory resistance.
The best-known example is DoNotPay, which marketed itself as “the world’s first robot lawyer.” In 2024, the US Federal Trade Commission fined the company $193,000 over deceptive claims and the production of substandard legal documents.
Such ventures frequently risk crossing the line into the unauthorized practice of law, a criminal offense in several US states, including California.
Nevertheless, newer startups such as Cimphony AI have continued trying to sell an “autopilot for legal needs.” Cimphony recently shut down following widespread criticism, having buried a directly contradictory disclaimer in its terms of service: “The company is not a law firm and does not provide legal advice.”
Is there still reason for hope?
Despite the dangers, artificial intelligence can offer a genuine lifeline to people handling minor or urgent legal matters, provided they use it responsibly and understand that ultimate responsibility for the text remains theirs alone.
The story of Lynn White, a 71-year-old Southern California woman, illustrates the potential. After losing an eviction case and facing homelessness without the money to hire an appeals lawyer, she turned to ChatGPT. She, too, encountered the system’s “hallucinations.”
To overcome the problem, White developed a careful method: She pitted artificial intelligence engines against one another, checking information produced by ChatGPT against the Perplexity search engine to confirm that every ruling it cited actually existed.
Armed with verified arguments, White won her appeal and had the judgment against her overturned.
“It feels like David and Goliath,” White said. “Artificial intelligence is the slingshot.”
The legal profession is changing rapidly, but these cases demonstrate that one principle remains unchanged: a machine can never relieve a person of individual responsibility or the obligation to verify the facts.