India’s Supreme Court warns courts on AI hallucinations after fake precedents case

India’s top court publicly calling out AI hallucinations is a watershed moment for how legal systems will treat generative tools. Until now, most AI‑in‑law stories have centered on individual lawyers misusing ChatGPT. Here, a tribunal itself appears to have relied on bogus authorities, forcing the Supreme Court to step in and lay down a zero‑tolerance line. That elevates hallucinations from an embarrassing bug to a systemic risk that can undermine judicial legitimacy.

In practice, this will push Indian courts and bar councils toward clearer rules on when and how AI‑assisted research can be used, and what disclosure and verification standards apply. Similar moves have already happened in the US and Europe, but India’s massive, overburdened court system is both more dependent on research shortcuts and more vulnerable to error. Expect local legal‑tech startups to race to offer “hallucination‑safe” tools with built‑in citation checking and curated corpora.

For AGI, the lesson is that even incremental reliability problems can have outsized social consequences when they touch high‑stakes domains. If frontier labs want their systems trusted in law, medicine or public administration, they will need to demonstrate not just capabilities but robust guardrails—and accept that courts may impose their own.