Recent searches
Search options
Administered by:
@ZenHeathen@beige.partyI would like a website that lists other websites, apps, etc. and lists whether or not they are known to scrape for AI, or specifically allow their users' content to be scraped. Has anyone seen one?
@ZenHeathen following to find out.
摏 @ZenHeathen Unfortunately, unless the operators of the site take *extensive* actions to block the AI scrapers, scrapers that ignore robots.txt AND try to act like legitimate website visitors, any site that does not lock content behind a login/paywall of some sort is subject to being indexed by these systems, independent of what the site's stated policies are.
It took me almost a month to block them from my sites, and I'm virtually certain that some are getting past my defences anyway.
@alan Yeah, I understand that for websites. They're out there scraping *everything*.
But then there are sites like Wordpress who cut deals to make on money on deliberately providing the data, while none of that money goes to the people who wrote the blogs.
But for something like, say, an Android notes app, it should only be available to LLMs if it's done specifically, and I don't trust that they'd tell users, or keep that notice available for potential new users to see.