About

How the tracker is built and checked

The starting point is the Australian rows of Damien Charlotin's AI Hallucination Cases database. Each judgment's text is extracted locally, with publisher headers and footers removed. Claude (Anthropic) is asked to quote the passages on the AI error, the impugned citations, consequences, referrals and practice notes cited, and every quote is then checked by script against the judgment text, character for character, with its paragraph number. A second step proposes field values from those verified quotes only, and any value without a supporting quote is discarded.

Every row then goes through automatic checks: the citation matches the judgment's header, titles and impugned citations are copied exactly, characterisations have textual support, paragraph counts match the court's own certificate, and the row is compared with Charlotin's coding. These checks prove that each value rests on real text at the stated paragraph. They cannot prove the text was read correctly, so rows that pass are labelled auto-verified. A random sample of rows, fixed before any extraction, plus every flagged row, is checked by hand against the official judgment and labelled spot-checked.

Sources and a pending question

Most judgment copies in Charlotin's database were printed from AustLII. Our code never accessed AustLII, but the text of those copies (with AustLII's headers removed) was processed by the AI model. We are asking AustLII whether this use is acceptable. Until it replies, the details the AI extracted from those copies are held back. Rows checked by hand against the judgment, facts read by script (citation, court, date) and Charlotin's own coding are shown.

Contributions and AI use

This project was built with Claude Code (Anthropic), an AI coding assistant, and this is disclosed in full. Claude wrote the pipeline, the site and first drafts of the rules summaries and checklist, and Claude models perform the first-pass extraction. Luc Taya designed the project, made the judgement calls recorded in the repository's decisions log, reviews the spot-check sample and flagged rows, checks the rules matrix and edits the checklist. No figure is typed by hand: each is produced by a committed script from the data.

Privacy

The tracker records only what each judgment says. Self-represented parties can be filtered but not searched by name, and their case titles are not shown on the tracker page. Family law matters use only the court's anonymised title (Family Law Act 1975 (Cth) s 114Q).

Corrections and takedown

Email ai.courts.corrections@proton.me. Requests are answered within 7 days, and every change is logged in the changelog. If you are a self-represented party named in a decision, you can ask for identifying detail to be removed even where the entry is accurate.

Licence and credits

Data CC BY 4.0. Built on Damien Charlotin's AI Hallucination Cases database (CC BY 4.0). Court instruments are linked to their official sources.

Luc Taya, Economics (Honours) graduate, University of Western Australia.