Made-up court rulings. Who is liable when AI gets it wrong?
Author: Andrzej Albera, Responsible for strategy and the company's direction
The situation. A procurement specialist at a construction company has to prepare an appeal to the National Appeals Chamber by tomorrow. She asks an AI chat for rulings to support her argument. She gets three case numbers, each with a short summary that reads like a textbook. The appeal goes out. At the hearing, the contracting authority's lawyer calmly points out that none of these rulings says what was attributed to it, and one of them is not in the database at all.
What the law says
- It is already happening in Polish courts. On 23 June 2026 the Supreme Administrative Court (NSA) dismissed an appeal by a business from Łódź in a dispute over more than PLN 211,000 of VAT. The lawyer cited three rulings that did not contain the holdings attributed to them, and the court was highly critical of a professional representative using AI tools without reflection (case I FZ 104/26). On 31 August the NSA checked seven rulings cited in another cassation appeal: some had wrong dates or concerned other matters, and some had never been registered (case I OSK 590/26).
- Worldwide there are more than 1,300 cases. That is how many instances of made-up or distorted rulings in court filings had been collected by April 2026 in the database run by Damien Charlotin of HEC Paris. In Australia, Deloitte refunded part of its fee for a government report that contained non-existent sources.
- The company is liable, not the chat provider. Towards the client, the company is liable for improper performance of a contract under the general rules (Article 471 of the Polish Civil Code). For damage caused by an employee to a third party in the course of their duties, the employer is liable (Article 120 of the Labour Code). An employee who acted unintentionally is liable towards the company for no more than three months' pay (Article 119). Professional lawyers also face disciplinary liability.
- EU law does not change this. The European Commission has withdrawn its proposal for an AI Liability Directive. The new Product Liability Directive will cover software and AI from 9 December 2026, but it protects against personal injury and damage to private property and data, not against a company's loss caused by a badly prepared filing.
- The AI Act requires skills. A company must make sure its people know how to use AI (Article 4). An employee who does not know that a chat can invent a case number does not have that skill yet.
What to do
- Adopt a rule: an AI answer is not a source. A case number, provision, figure or name from an AI answer goes into a document only after it has been checked against the source: a case law database, the Journal of Laws, a company document. One sentence added to the AI usage rules We wrote about in the post AI inspections and fines from 28 October is enough.
- Name who signs. Every document that leaves the company has an author who checked it and is responsible for it. That person, not the tool, answers to the client, the court or the authority.
- Give people a tool that shows its source. A general-purpose chat answers from the model's memory, which is why it can make things up. An assistant that answers from company documents and gives a source with every answer lets people check the result in seconds.
This is how our SAIE ecosystem works. TWIN:DESK is a Digital AI Desk where assistants answer from the organisation's knowledge and give the source of every answer, so checking a case number or a contract clause takes one click. PROXY:AI keeps a register of AI use, so when a complaint comes in you can see who asked the model what, and when. The CDF methodology introduces the Human Competence Gate: a person approves an AI output only after showing that they understand it.
Frequently asked questions
Can you sue the chat provider for a made-up answer?
In practice, it is difficult. The terms of popular tools state that answers may contain errors and must be checked. Liability towards the client stays with the company.
Will the employee pay for an AI error?
The company is liable towards the client. An employee who acted unintentionally is liable towards the employer for no more than three months' pay. A clear duty to check, and a tool that makes checking easy, work better than penalties.
Why does AI make things up?
A language model produces the most probable sequence of words. A case number in the correct format looks credible, even if something completely different was decided in that case. The model does not check a case law database unless someone gives it access and tells it to rely on it. We describe how we limit hallucinations on our page Technologies.
Don't paid legal tools have this problem?
They have it less often, because they use case law databases, but they are not free of it. A 2024 Stanford University study found incorrect answers in one in six or more queries to specialised legal research tools. We wrote more about AI in law firms in the post Does a lawyer need sovereign AI?
Is it enough to add a note that the filing was prepared with AI?
No. Such a label does not remove liability. The court and the client assess the content, and the person who signed the document is responsible for it.
The opportunity
AI that gives its source cuts the search for rulings, provisions and contract clauses from hours to minutes. This gain stays in the company only when checking is built into the way people work rather than left to goodwill. Then AI stops being a risk and becomes an advantage.