LIVE · scanned from real robots.txt

The independent web is defenseless against AI.

We scanned the robots.txt of 110 independent US sites — bloggers, newsletters, data creators. Most aren't blocking AI. And none of them are getting paid for it.

The split
Open — no AI rules at all. Giving it away. Managing — curating which bots get in. Blocking — walling AI out (but earning $0).
72%
are wide Open — not blocking a single AI crawler. This is the money left on the table.
17%
are Managing — already curating AI by bot. Exactly who our per-purpose pricing is built for.
11%
are Blocking — they slammed the door. We say: charge instead.
Most-blocked AI crawlers
Among sites that block anything, OpenAI's GPTBot is the most-blocked crawler — the lightning rod of the AI-scraping backlash. Every bot here is one your content feeds for free today.
Every site we scanned
SiteStatusBlocks
Methodology. One-time scan of each site's public /robots.txt (Aug 2026); rescanned periodically, not at page load.txt, checking 13 named AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider, Meta-ExternalAgent, and more). "Blocking" = disallows 5+ of them from /; "Managing" = names 1–4; "Open" = names none. Sample of independent US publishers, August 2026. robots.txt is a request, not a lock — but it's the clearest public signal of intent. A few large news sites weren't reachable from our scan environment and are excluded.

Blocking is losing. Charging is winning.

72% giving it away, 11% walling it off — and 0% getting paid. Tolljar is the third option: let AI in, and make it pay. Put a meter on your work in 3 minutes.

🫙 Join the waitlist →