Crawling settings

One add-on that decides whether search engines, AI crawlers, and everything else may crawl your archive, separate from who can read it.

Justin Duke

September 12, 2026

Some folks want their archives available so they can link to them, but don't want them to come up in Google results.

Some folks want to prohibit LLMs and other scrapers from crawling their archives.

Both of these are eminently reasonable! We tried to handle a lot of this with a very engineer-brained approach: give you a big ol' textarea where you can type user agent strings to block.

People do not know or care about user agent strings, we realized. So we replaced all of it with a Crawling add-on in Settings → Archives, holding one toggle per kind of crawler.

demo.buttondown.com/settings/archives

This is a live demo. You can view this page on our live demo site, too.

We looked at usage (and our support volume) and found the two biggest reasons/avenues that y'all care about, plus a catch-all for everyone else:

Toggle	Examples
Search engines	Googlebot, Bingbot, DuckDuckBot, Applebot
AI crawlers	GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and a few dozen more
Everything else	Scrapers, SEO tools like AhrefsBot, archivers, and bots that don't say who they are

The docs have the full rundown of what each toggle does under the hood.

Frequently asked questions

What actually changed for me?

For almost everyone, nothing. All three toggles default to on, which matches how your archive already behaved: search engines could index it and AI crawlers could read it. The only newsletters whose behavior changed are the few dozen that had set one of the old hidden noindex flags or typed entries into the blocked user agents box; those were migrated to the matching toggle so they keep doing what they did before.

What happened to blocked user agents?

If you had entries in the old blocked user agents textarea, we migrated you. A list that named AI crawlers becomes "AI crawlers: off". A list that named Googlebot or Bingbot becomes "Search engines: off". A list that named anything else — vulnerability scanners, SEO bots, a stray archiver — becomes "Everything else: off", which disallows every crawler outside the two named buckets. If you were relying on very fine-grained per-agent rules, the firewall is a better home for them than robots.txt.

Is this paywalled?

No. The whole add-on is available on every plan, including the free one. The only wrinkle is that fully enforcing the AI crawlers and everything else toggles through robots.txt needs a custom hosting domain, which is a paid feature; without one, those toggles still block sign-ups from AI crawlers and tag every response, but can't write to robots.txt.

Buttondown is the last email platform you’ll switch to.

Join your last newsletter platform