How AI companies are secretly collecting training data from the web (and why it matters)

Getty/the_burtons

Like astir people, my woman types a hunt into Google galore times each day. We enactment from home, truthful our household country doubles arsenic a league room. Whenever we're successful a meeting, and a question astir thing comes up, she Googles it.

This is the aforesaid arsenic it's been for years. But what happens adjacent has changed.

Instead of clicking connected 1 of the hunt effect links, she much often than not reads the AI summary. These days, she seldom clicks connected immoderate of the sites that supply the archetypal accusation that Google's AI summarizes.

Also: How overmuch vigor does AI truly use? The reply is astonishing - and a small complicated

When I spoke to her astir this, Denise acknowledged that she really visits sites little frequently. But she besides pointed retired that, for topics wherever she's well-versed, she has noticed the AI is sometimes wrong. She said she takes the AI results with a atom of salt, but they often supply basal capable accusation that she needs to look nary further. If successful doubt, she does excavation deeper.

So that's wherever we are today. More and much users are similar my wife, getting information from the AI and ne'er visiting websites (and truthful ne'er giving contented creators a accidental to beryllium compensated for their work).

Worse, much and much radical are trusting AI, truthful not lone are they making it harder for contented creators to marque a living, but they are often getting hallucinatory oregon incorrect information. Since they ne'er sojourn the archetypal sources of information, they person small impetus to cross-check oregon verify what they read.

The interaction of AI scraping

Cloudflare CEO Matthew Prince offered some devastating statistics. He utilized the ratio of the fig of pages crawled compared to the fig of pages fed to readers arsenic a metric.

As a baseline, helium said that 10 years ago, for each 2 pages Google crawled, it sent 1 visitant to a contented creator's site. Six months ago, that ratio was six pages crawled to 1 visitant sent to a contented site. Now, conscionable six months later, it's 18 pages crawled to 1 visitant sent to a contented site.

The numbers, according to Prince, are acold worse for AI sites. AI sites deduce important worth from accusation they've scraped from each the remainder of us. Six months ago, the ratio of pages scraped to visitors redirected via OpenAI was 250 to 1. Now, arsenic radical person go much acquainted with trusting (or being excessively lazy to attraction astir inaccuracies), the ratio is 1,500 to 1.

In galore ways, AI is becoming an existential menace to contented creators. By vacuuming up contented produced by hard-working teams each crossed the world, and past feeding that contented backmost arsenic summaries to readers, the publishers and writers are losing gross and influence. Many creators are besides losing motivation, due to the fact that if they can't marque a surviving doing it, oregon astatine slightest make a following, wherefore bother?

Also: AI agents volition endanger humans to execute their goals, Anthropic study finds

Some publishers, similar Ziff Davis (ZDNET's genitor company) and the New York Times, are suing OpenAI for copyright infringement. You've astir apt seen the disclaimer connected ZDNET that says, "Disclosure: Ziff Davis, ZDNET's genitor company, filed an April 2025 suit against OpenAI, alleging it infringed Ziff Davis copyrights successful grooming and operating its AI systems."

Other publishers, including the Wall Street Journal, the Financial Times, the Atlantic, and the Washington Post, person licensed their contented to OpenAI and immoderate different AI ample connection models.

The harm to nine arsenic a full that AI intermediation tin origin is profound and worthy an nonfiction each connected its own. But this nonfiction is much practical. Here, we admit the menace AI presents to publishing, and absorption connected method ways to combat back.

In different words, if the AIs can't scrape, they can't springiness distant published and copyrighted contented without publishers' permission.

Robots.txt: Your archetypal defence

The simplest, astir direct, and perchance slightest effectual defence is the robots.txt file. This is simply a record you enactment astatine the basal of your website's directory. It tells spiders, crawlers, and bots whether they person support to entree your site. This is besides called User-Agent filtering.

This record has a fig of absorbing implications. First, lone well-behaved crawlers volition wage attraction to its specifications. It doesn't supply immoderate information against access, truthful compliance is wholly voluntary connected the portion of the bots.

Also: 15 caller jobs AI could make - could 1 beryllium your adjacent gig?

Second, you request to beryllium cautious which bots you nonstop away. For example, if you usage robots.txt to contradict entree to Googlebot, your tract won't get indexed for searching connected Google. Say goodbye to each Google referrals. On the different hand, if you usage robots.txt to contradict entree to Google-Extended, you'll artifact Gemini from indexing and utilizing your tract for Gemini training.

This site has an scale of those bots you mightiness privation to contradict entree to. This is OpenAI's guide connected however to forestall OpenAI's bots from crawling your site.

But what astir web scrapers that disregard robots.txt? How bash you forestall them from scraping your site?

How tin you forestall rogue scraping?

It's present that tract operators request to usage a belts-and-suspenders strategy. You're fundamentally successful an arms contention to find a mode to support against scraping, portion the scrapers are trying to find a mode to suck down each your site's data. In this section, I'll database a fewer techniques. This is acold from a implicit list. Techniques alteration constantly, some connected the portion of the defenders and the scrapers.

Rate bounds requests: Modify your server to bounds however galore pages tin beryllium requested by a fixed IP code successful a play of time. Humans aren't apt to petition hundreds of pages per minute. This, similar astir of the techniques itemized successful this section, volition disagree from server to server, truthful you'll person to look up your server to find retired however to configure this capability. It whitethorn besides annoy your site's visitors truthful overmuch that they halt visiting. So, there's that.

Use CAPTCHAs: Keep successful caput that CAPTCHAs thin to inconvenience users, but they tin trim immoderate types of crawler entree to your site. Of course, the irony is that if you're trying to artifact AI crawlers, it's the AIs that are most apt to beryllium capable to decision the CAPTCHAs. So there's that.

Selective IP bans: If you find determination are IP ranges that overwhelm your tract with entree requests, you tin prohibition them astatine the firewall level. FireHOL (an unfastened root firewall toolset) maintains a blacklist of IP addresses. Most of them are cybersecurity-related, but they tin get you started connected a artifact list. Be careful, though. Don't usage broad IP bans, oregon morganatic visitors volition beryllium blocked from your site. So, there's that, too.

Also: 5 ways you tin plug the widening AI skills spread astatine your business

The emergence of anti-scraping services

There are a increasing fig of anti-scraping services that volition effort to support your tract for a fee. They include:

• QRATOR: Network-layer filtering and DDoS-aware bot blocking
• Cloudflare: Reputation-tracking, fingerprinting, and behavioral analysis
• Akamai Bot Manager: Identity, intent, and behavioral modeling
• DataDome: Machine learning positive real-time response
• HUMAN Security: JavaScript sensors with Al backend
• Kasada: Adaptive challenges and alleged tamper-proof JavaScript telemetry
• Imperva: Threat quality positive browser fingerprinting
• Fastly: Rule-based filtering with borderline logic
• Fingerprint: Cross-session fingerprinting and idiosyncratic tracking
• Link11: Behavioral investigation and postulation sandboxing
• Netacea: Intent-based detection and server-side analytics

Here's a speedy overview of immoderate of the techniques these services use.

Behavior matching: This method analyzes much than headers; it analyzes petition behavior. It's fundamentally a operation of header investigation and bot-by-bot petition limiting.

JavaScript challenges: Beyond JavaScript-based CAPTCHA, these often tally successful the inheritance of a web page. They necessitate scripts to execute oregon measurement the pacing of enactment connected the leafage to let further access.

Honeypot traps: These are often elements buried successful a web page, similar invisible fields oregon links, that are designed to seizure bots. If a bot grabs everything connected a tract (which a quality idiosyncratic is improbable to do), the honeypot trap recognizes it and initiates a server block.

Overall behavioral analysis: This is wherever AIs are warring AIs. AIs moving connected behalf of your website show entree behavior, and usage instrumentality learning to place entree patterns that are not human. Those malicious accesses tin past beryllium blocked.

Browser fingerprinting: Browsers supply a wide scope of information astir themselves to the sites they access. Bots mostly effort to spoof the fingerprints of morganatic users. But they often inadvertently supply their ain fingerprints, which blocking services tin aggregate and past usage to artifact the bots.

Decoy traps: These are mazes of decoy pages filled with autogenerated and useless content, linked unneurotic successful a signifier that causes bots to discarded their clip oregon get stuck pursuing links. Most of those are tagged with "nofollow" links, truthful hunt engines don't scale them oregon negatively impact your SEO rank. Of course, malicious bots are learning however to place these traps and antagonistic them, but they bash connection constricted protection.

The large trade-off of blocking scraping for AI grooming

As an writer who makes my surviving straight from my originative output, I find the imaginable of AIs utilizing my enactment arsenic grooming information to beryllium offensive. How situation a institution similar OpenAI marque billions disconnected the backs of each of america creatives! They past crook astir and supply a merchandise that could perchance enactment galore of america retired of work.

And yet, I person to admit that AI has saved maine clip successful galore antithetic ways. I usage a substance exertion oregon a connection processor each day. But backmost erstwhile I started my career, the publications I wrote for had typesetting operators who converted my written words into publishable content. Now, the blogging tools and contented absorption systems bash that work. An full assemblage vanished successful the abstraction of a fewer years. Such is the terms of caller technology.

I've been progressive with AI innovation for decades. After penning astir generative AI since it boomed successful aboriginal 2023, I'm convinced it's present to stay.

Also: The astir captious occupation accomplishment you request to thrive successful the AI revolution

AI chatbots similar Google Gemini and ChatGPT are making token efforts to beryllium bully citizens. They scrape each our contented and marque billions disconnected of it, but they're consenting to supply links backmost to our enactment for the precise fewer who fuss to cheque sources.

Some of the large AI companies contend that they supply worth backmost to publishers. An OpenAI spokesperson told Columbia Journalism Review, "We enactment publishers and creators by helping 400M play ChatGPT users observe prime contented done summaries, quotes, wide links, and attribution."

Quoted successful Digiday, David Carr, elder insights manager astatine information analytics institution Similarweb, said, "ChatGPT sent 243.8 cardinal visits to 250 quality and media websites successful April 2025, up 98% from 123.2 cardinal visits this January." 

Those numbers are big, but lone without context. Google gets billions of visits a day, and earlier AI, astir each those visits resulted successful referrals retired to different sites. With Google's referral percentages dropping precipitously and OpenAI's referral numbers being a precise tiny percent of postulation different sent to contented producers, the occupation is precise real.

Yes, those links are specified array scraps, but bash we artifact them? If you alteration web scraping blocks connected your website, volition it bash thing different than "cut disconnected your chemoreceptor to spite your face," arsenic my parent utilized to say?

Also: Sam Altman says the Singularity is imminent - here's why

Unless each tract blocks AI scrapers, efficaciously locking AI information sets to 2025 and earlier, blocking your ain tract from the AIs volition execute small much than preventing you from getting what small postulation determination is from the AI services. So should you?

In the agelong term, this signifier of AI scraping is unsustainable. If AIs forestall creatives from deriving worth from their hard work, the creatives won't person an inducement to support creating. At that point, the prime of the AI-generated contented volition statesman to decline. It volition go a vicious circle, with less creatives capable to monetize their skills and the AIs providing ever-worsening contented quality.

So, what bash we bash astir it? If we are to past into the future, our full manufacture needs to inquire and effort to reply that question. If not, invited to Idiocracy.

What astir you? Have you taken immoderate steps to artifact AI bots from scraping your site? Are you acrophobic astir however your contented mightiness beryllium utilized to bid generative models? Do you deliberation the trade-off betwixt visibility and extortion is worthy it? What kinds of tools oregon services, if any, are you utilizing to show oregon bounds scraping? Let america cognize successful the comments below.

You tin travel my day-to-day task updates connected societal media. Be definite to subscribe to my play update newsletter, and travel maine connected Twitter/X astatine @DavidGewirtz, connected Facebook astatine Facebook.com/DavidGewirtz, connected Instagram astatine Instagram.com/DavidGewirtz, connected Bluesky astatine @DavidGewirtz.com, and connected YouTube astatine YouTube.com/DavidGewirtzTV.