Meta has scraped data from the most-trafficked domains on the internet —including news organizations, education platforms, niche forums, personal blogs, and even revenge porn sites—to train its artificial intelligence models, according to a leaked list obtained by Drop Site News.
By scraping data from roughly 6 million unique websites, including 100,000 of the top-ranked domains, Meta has generated millions of pages of content to use for Meta’s AI-training pipeline.
The sites that Meta scrapes consist of copyrighted content, pirated content, and adult videos, some of whose content is potentially illegally obtained or recorded, as well as news and original content from prominent outlets and content publishers.
They include mainstream businesses like Getty Images, Shopify, Shutterstock, but also extreme pornographic content, including websites advertising explicit sexual content and humiliation porn that exploits teenagers.
% grep lemmy Meta.txt lemmy.ca lemmynsfw.com lemmy.sdf.org lemmy.ml lemmy.world lemmygrad.mlwhere did you get the .txt file?
Just extracted it from the PDF
Good catch. That’s worth a seperate post.
Hexbear is on the list too.
Llms will start randomly shitting out hexbear emotes lol
One person’s “scraping” is another person’s plagiarism.
We ban the petty scrapers and celebrate the great ones as innovators and promote them as fortune 500 companies.
I blocked the entire ASN for Meta, because they were downright dirty with their scraping. No gradual crawling, fakes UAs, random addresses across a large number of subnets.
They weren’t the only ones either. The AI scraping heist is the new goldrush.
My Mastodon instance is on the list. I try hard to block them.
The problem with the list is that it’s a target list, but not a list showing how much content, if any, they manage to process from any of the sites.
Stole.
Always amused when leftist instances treat intellectual property like it’s real.
its not, but scraping is annoyingly resource intensive.
IP debate aside, LLM scrapers absolutely annihilate system resources. I host a wordpress site and before setting up cloudflare labyrinth my whole server would get ddos’d at least twice a day.
