The Italian data protection authority Garante per la protezione dei dati personali has issued the US company Luka Inc. with a Fine sanctioned in the amount of 5 million euros. Luka Inc. allegedly violated central data protection principles of the GDPR with its AI-supported chatbot Replika. GDPR have violated the law. The decision is remarkable not only because of the amount of the sanction, but also because of its detailed legal examination of the challenges posed by generative AI systems and the protection of minors.
Replica and its risks
Replika is an AI-supported chatbot that can act as an emotional companion, friend or romantic partner via text-based and voice interaction. The system was marketed in particular to promote emotional well-being. It also appeals to mentally stressed or socially isolated people.
Replika is technically based on a so-called Large Language Model (LLM). Its performance is improved through continuous interaction with users. This further development is carried out using real user data, including personal information such as communication content, vocal expressions and, where applicable, information on mental state or emotional condition. This is particularly sensitive from a data protection perspective, as such data is often particularly sensitive and allows conclusions to be drawn about the personality, moods and habits of the people concerned.
Core of the allegations against Luka Inc.: three serious data protection violations
Lack of legal basis for data processing (Art. 5 para. 1 lit. a, Art. 6 GDPR)
The privacy policy valid at the time of the investigation (version dated July 5, 2022) did not show a differentiated allocation of Legal basis to specific processing operations. In particular the Processing for the further development of the LLM (Model Development) was made without a sufficient legal basis. This violates the principle of lawful, fair and transparent development. Processing.
Insufficient Duty to inform (Art. 12, 13 GDPR)
The privacy policy was only available in English, which violates Art. 12 para. 1 sentence 2 GDPR violates. The standard requires information to be provided in clear and easily understandable language - including for Italian-speaking users and minors. In fact, the privacy policy did not contain any information on the storage period or clear information on the use of personal data, transfers outside the EU or the Processing through automated decision-making. This resulted in a considerable lack of transparency regarding the type and scope of data processing.
Lack of protection of minors (Art. 5 para. 1 lit. c, Art. 24, 25 GDPR)
Despite explicitly addressing the target group (emotional support, romantic relationship), Replika did not contain any effective mechanisms for age verification. Minors were able to use the service unhindered, even if they openly identified themselves as being under 18. In particular, the data protection supervisory authority criticized the lack of implementation of technical protective measures such as a robust "age gate", an effective "cooling-off" procedure or voice-based detection methods.
Sanctions against replicas: amount, justification and further measures
The Italian data protection authority imposed a fine on Luka Inc. Fine in the amount of 5 million euros. The basis for this sanction was Art. 83 para. 5 GDPRwhich provides for the highest fines for particularly serious violations of key data protection principles. When calculating the specific amount, the authority also took into account the limits set out in Art. 83 Para. 2 GDPR mentioned criteria. These include the type, severity and duration of the infringement, the number of persons affected, the extent of the damage caused and the conduct of the controller during the proceedings.
In this case, several aggravating circumstances were identified: First, there was a structural, systematic disregard of basic data protection requirements - in particular with regard to Transparency, Earmarking, Legal basis and technical protection measures. Secondly, a large number of users were affected, including vulnerable groups such as minors. Thirdly, the international reach of Replika posed a significant risk to EU citizens, as adequate safeguards were not in place to protect cross-border data processing.
In addition to the Fine the Italian authority obliged Luka Inc. to make specific improvements. These included the provision of a complete, easy-to-understand privacy policy in Italian. In addition, the information on data transfers to third countries and on storage periods and processing purposes must be made much more transparent and GDPR-compliant.
There is also a particular focus on implementing an effective age verification system. This should effectively control access for minors in future and ensure age-appropriate safeguards.
Reading tip: AI competencies and company obligations according to Art. 4 AI Regulation
Practical significance: precedent for AI services in the EU
The decision of the Italian data protection authority against Luka Inc. has fundamental significance for the handling of generative AI systems within the European Union beyond the individual case. It makes it clear that the GDPR also applies without restriction to highly dynamic, technology-driven business models. Even if the responsible provider has its registered office outside the EU. This is because the market place principle (Art. 3 para. 2 GDPR) ensures the applicability of European data protection law wherever personal data processed by persons resident in the EU.
This sends a clear signal for data protection practice: AI applications must be designed from the outset in such a way that Data protection and Data security be taken into account structurally. This applies in particular to business models that specifically target emotional or psychological aspects of user behavior. The consideration of protection mechanisms for minors, transparent information about automated decision-making processes and the clear delimitation of the purposes of data processing are absolutely essential.
The Replika case also shows that the supervisory authorities are prepared to subject even technically complex systems to a thorough legal analysis and impose severe sanctions if violations are found. This should also encourage other providers of chatbots, virtual assistants or digital therapy services to subject their processes and data protection declarations to a critical review.
Last but not least, the decision is also an appeal to international regulation: While the GDPR already established a high standard of protection, the case illustrates the need for global minimum standards for Transparencyresponsibility and property rights in the field of artificial intelligence. The Luka Inc. case could therefore play a pioneering role for further proceedings at national and European level and serve as a blueprint for a data protection-compliant AI ecosystem in Europe.
German 
English 
Italian 
French