🤖 Ghosts in the Machine: Why AI Hallucination Breaks Privacy Laws
Language models are not databases, and privacy laws were not made for them.
We are facing a regulatory impasse that demands a new approach, one focused on outputs, not on "fixing" the impossible.
In my view, the greatest challenge of generative AI isn't technical, it's ontological. We are trying to apply the Newtonian physics of privacy and data protection to a quantum universe of probabilities, and the result is a regulatory paradox.
Hallucination, the generation of convincing yet false information, is not a bug but an intrinsic feature of how LLMs work. They don't store facts in records; they process text into abstract mathematical representations (tokens, embeddings) and predict the next most likely word based on statistical patterns.
On the other hand, we have the rigidity of regulations like the GDPR, which demands absolute data accuracy and grants the right to rectification. The case by NOYB against OpenAI, concerning an incorrect birthdate generated by ChatGPT, is the perfect example of this clash of realities.
The law requires a correction that, technically, is like asking to remove a specific memory from the human brain without surgery. Rectification in an LLM isn't like editing a spreadsheet cell; it would require prohibitively expensive and complex retraining, something that OpenAI itself admits is unfeasible in all cases. And not just OpenAI, it applies to practically any model from any company.
The right to rectification, when interpreted as an "obligation of result," creates an existential threat for current models, as technical impossibility is not a valid excuse.
Recently, a case involving ChatGPT and the Hamburg data protection authority (HmbBfDI) highlighted the complexity of applying the GDPR's accuracy principle to LLM hallucinations. The German authority, instead of focusing on the model's training data, adopted a more pragmatic, risk-focused approach. They argued that the analysis should concentrate on the AI-generated result (the output) and its context of use. This distinction is important, as it acknowledges the probabilistic nature of LLMs and shifts responsibility to how the generated information is presented and used, rather than attempting a technically infeasible correction within the model itself.
To me, this seems the most logical.
Regulation should focus on the actual harm, which occurs at the output—the point of contact with the user. After all, when we talk about privacy laws, we are fundamentally talking about ensuring that the use of personal data respects the purposes for which that data was collected. If the output respects that purpose, does it really matter if a set of algorithms used that data to generate mathematical probabilities within a model?
Combined with the risk-based approach from the UK's ICO, which assesses the need for accuracy based on the application's purpose, we have a more technical and innovation-friendly path forward. For us, technology professionals in Brazil, I believe that waiting for an AI Act is a reactive and dangerous strategy. The LGPD (Brazil's General Data Protection Law) already gives us the principles. We must proactively implement output controls, such as filters and RAG (Retrieval-Augmented Generation), be transparent about the models' limitations, and create effective DSAR (Data Subject Access Request) channels to manage the outputs.
In addition to managing inherent hallucinations, we need to protect models from external manipulation. Preventing techniques like prompt injection, where an attacker can deliberately force the LLM to generate misinformation or leak data, becomes a critical front line. It's not enough to deal with the internal "ghosts" in the code; we have to build robust walls against the "invaders" trying to control them. Effective AI governance requires a defense-in-depth approach, combining output risk management with stringent input security.
We can't "fix" hallucination, but we can and must architect systems of trust around it, don't you think?
Compliance in the AI era won't be about having perfect data, but about managing imprecision responsibly (I think I'll frame that sentence).
Have a great week!