European non-profit AI Forensics found an "ecosystem of abuse at scale" operated on Telegram, with users distributing and selling harmful sexual content across borders despite the platform’s moderation efforts.
A large-scale network of just under 25,000 people spread nonconsensual sexual material and child pornography throughout Spain and Italy on the instant messaging service Telegram, according to a new report.
ADVERTISEMENT
ADVERTISEMENT
AI Forensics, a European non-profit institute that studies algorithms, found an "ecosystem of abuse at scale" after reviewing 2.8 million Telegram messages across 16 groups over six weeks.
According to its report, most of the users sharing nonconsensual sexual content are young, heterosexual men sharing content of women that are their partners, former partners or acquaintances. The most extreme content depicts children in incest and rape scenarios.
The report comes after members of the European Parliament backed a ban on artificial intelligence (AI) systems that "nudify" images — or create fake nude images — by removing clothing from a subject without their consent last month.
One in three women in the European Union has experienced sexual violence since the age of 15, which includes cyberviolence, according to the European Institute for Gender Equality.
What content is being spread?
The study found that the young men often obtain the source material from private exchanges on the social media application Instagram and the private messaging application WhatsApp.
The perpetrators monetise the videos by charging one-time fees of up to €50 for full access to channels and archives of this content. In some cases, they also charge a monthly subscription fee of €5.
"Nudifying" bots are often advertised on those channels as tools to generate more nonconsensual content.
"This has allowed perpetrators to scale up the volume, speed and ease at which [this content] is perpetrated," according to an AI Forensics press release.
The women featured in the videos are often named, tagged, and can be located through profile links that are shared in Telegram channels, the study found.
This content is then redistributed on other platforms, such as TikTok and Instagram. Reddit is used as a "recruitment gateway" that disseminates links to the original Telegram channels where people can find the paid content.
The content reached approximately 52,000 people: 27,000 people in Italy and another 25,000 in Spain, AI Forensics found.
The NGO said this suggests that the way nonconsensual sexual content is being spread on social media is not restricted by national borders or platform boundaries and is a "structural problem that is European in scope and demands a European response."
Telegram moderation 'insufficient', report says
Telegram reportedly shut down some of the groups that AI Forensics observed, but these were created again with the same names "just a few hours later." The European organisation said that means Telegram’s moderation practices are "insufficient".
It recommended that moderation on Telegram should include reporting mechanisms and enforcing the platform’s policies, especially the Premium subscription model, because it gives creators a way to monetise this content.
The non-profit said the European Commission should also designate Telegram as a Very Large Online Platform (VLOP) under the Digital Services Act (DSA), the EU's legal framework for online platform accountability, content moderation, and user safety.
This designation would mean the platform will be subjected to stricter obligations, such as being transparent about how its algorithm works and conducting risk assessments for harmful content.
In February, the platform claimed that it had "significantly fewer" than 45 million users, which is the EU’s cutoff for being labelled a VLOP.
AI Forensics also wants to see the EU AI Act include provisions to make it easier to remove child sexual abuse and nonconsensual images from platforms.
Euronews Next contacted Telegram for a response to the report, but did not receive an immediate reply.
Child sexual content and non-consensual material are restricted on Telegram, according to the platform’s terms of service. AI-powered content moderation, as well as staff, are in place to counter abuse. Users can also report content that violates the platform’s terms and conditions.
Telegram can temporarily or permanently suspend certain functions on someone’s account, such as communicating with people they are not connected to.
In cases of "serious violations", the platform could block or delete users, bots, posts, channels and groups.
