robots.txt and AI Crawlers: A Practical Guide for Small Business Websites
Imagine you publish a detailed pricing guide, a set of product descriptions, or a local-expertise blog. Then you read that AI companies may collect public web pages to train their models, and you wonder whether you can say no. Many site owners ask the same question, whether they run a shop in Texas, a consultancy in Pune, or a studio in Dhaka.
The tool most people reach for is a small text file called robots.txt. It is useful, but it is often misunderstood. This guide explains what it can do, what it cannot do, how Google's AI-related control works, and a short routine you can follow. It is technical guidance, not legal advice.
Table of contents
Executive summary
robots.txt is a public file that asks well-behaved crawlers which parts of your site they may fetch. It is a request, not a lock. Google offers a separate robots.txt token, Google-Extended, that controls whether content Google has crawled can be used for Gemini training and grounding, and it does not affect your place in Google Search. A shared industry signal is being drafted at the IETF, but it is not finished. For crawlers that ignore your file, you need server-side protection as a second layer.
Quick answer: Yes, you can state your preferences for AI crawlers by adding rules to the robots.txt file at the root of your domain. Reputable crawlers that honor the protocol will follow them, but the file does not technically stop anyone. Use it to express preferences, and add a firewall or bot management layer if you need enforcement.
What robots.txt actually does
The Robots Exclusion Protocol is specified in RFC 9309. The file must be named robots.txt, sit at the top level of your site (for example, https://www.example.com/robots.txt), be UTF-8 text, and be served as text/plain.
Rules are grouped by crawler name. Each group starts with a User-agent line, followed by Allow and Disallow rules that match the start of a URL path. If more than one rule matches, the longest one wins, and a crawler with no matching group falls back to the * group.
The RFC is also clear about limits: these rules "are not a form of access authorization." The file is public, so anyone can read the paths you list. Google's own robots.txt introduction adds that the file is mainly meant to manage crawl load and is not a way to keep a page out of Google. It also warns that combining crawling and indexing rules can cause some rules to counteract others.
Two different goals are easy to mix up. Keeping a page out of search results is an indexing question, and noindex is the tool for it. But Google's guidance on blocking indexing says the page must not be blocked by robots.txt and must be otherwise accessible to the crawler. If a Disallow rule blocks the page, the crawler never sees the noindex rule, and the URL can still appear in results. So do not pair Disallow with noindex as a removal method.
Restricting who can open a page is an access question, and neither robots.txt nor noindex answers it. Use authentication, such as password protection, for anything that must stay private.
Google crawlers and the Google-Extended token
Google's crawler overview separates common crawlers, special-case crawlers, and user-triggered fetchers. Common crawlers such as Googlebot always respect robots.txt for automatic crawls, while special-case crawlers can behave differently under agreements with the sites they serve.
Google-Extended is different from all of these. According to Google's common crawlers documentation, it is a robots.txt product token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no HTTP user agent of its own, it does not govern crawling, and it does not affect inclusion in Google Search or act as a ranking signal.
That last point matters for small businesses. Declining Google-Extended is a content-use choice, and Google states it is separate from search visibility. Blocking Googlebot itself is a very different decision with real consequences for how people find you.
The snippet below shows only the group to add or merge. It is not a complete robots.txt file and must not replace yours. Back up your current file first, keep your existing restrictions and any Sitemap lines, and check whether a Google-Extended group already exists. If one does, edit that group instead of adding a second one. Also review existing Allow and Disallow rules that match the same paths, because under RFC 9309 the longest matching rule wins and an equally specific Allow beats a Disallow.
# ...your existing rules and Sitemap lines stay as they are...
# Group to add, or merge into an existing Google-Extended group:
User-agent: Google-Extended
Disallow: /
# ...your existing rules and Sitemap lines stay as they are...
A group addressed to Google-Extended does not touch normal search crawling, since the token does not govern crawling.
What is coming: shared AI preference signals
Today each company publishes its own crawler names, so rules are per company. The IETF's AI Preferences working group is drafting a common vocabulary and a way to attach preferences to content, and it leaves out enforcement.
The attachment draft,draft-ietf-aipref-attach (version 05, August 2026), shows a Content-Usage rule in robots.txt and as an HTTP response header. Its example uses train-ai=n to say content should not be used for AI training. That syntax belongs to the draft, which can change, so do not deploy it as a standard.
Google's documentation describes something separate: its common crawlers page says Google intends to support the AIPREF shared training preference signal, written there as user-agent: __usage_ai-training, in early 2027. That page does not mention Content-Usage, so nothing there ties Google's plan to the draft's syntax. Google's Search Central Blog also lists an October 2026 post titled Supporting a shared AI training control for websites.
Both are planned or in draft; neither is available today. For now, rely on existing robots.txt tokens.
Which control does what
| Control | What it does | Limits |
|---|---|---|
| robots.txt Disallow | Asks crawlers that honor the protocol to skip certain paths | Not access control; the file is public and can be ignored. Blocking a page here can stop a crawler from seeing its noindex rule |
| Google-Extended token | Controls Gemini training and grounding use of content Google crawled | Google only; does not change Google Search inclusion or ranking |
| Content-Usage (IETF draft) | Proposed shared way to state AI usage preferences | Draft, not final; Google's documentation does not mention it, and support by crawlers varies and is not guaranteed |
| noindex rule | Asks Google to leave a crawlable page out of search results | The page must not be blocked by robots.txt or the rule is never seen; it does not restrict access |
| Password or other authentication | Restricts who can open a page | Needs setup, and visitors must log in; it is the option here that actually restricts access |
| Firewall or bot management | Blocks or rate-limits unwanted automated traffic at the server | Needs configuration and review so real visitors and search engines are not blocked |
A simple routine for your site
You do not need a long blocklist. Work through these steps once, then revisit them a couple of times a year.
- Read your current file: open
yourdomain.com/robots.txtin a browser. A plugin, theme, or host may already generate rules you did not write. - Back up and merge, never replace: save a copy of the current file, keep its existing restrictions and Sitemap lines, check for an existing
Google-Extendedgroup and any matchingAlloworDisallowrules, then edit or merge that group instead of overwriting the whole file. - Decide your policy: pick what you want to allow, limit, or keep private, and write it in a sentence before touching the file.
- Use official names: copy each crawler's token from that company's own documentation. Names change, so do not rely on old lists.
- Protect private pages properly: use passwords or access controls for anything sensitive, not a Disallow line.
- Test the result: confirm the file loads, returns a normal success status, and does not block your whole site by accident.
- Add a second layer if needed: use firewall or bot management rules for traffic that ignores robots.txt, and watch your server logs for unusual patterns.
Common mistakes to avoid
Blocking everything with one line
A Disallow: / under User-agent: * asks every crawler to stay away, including search engines. Check that any broad rule is intentional before you save it.
Treating robots.txt as security
Listing a private folder in a public file can point people to it. Hide sensitive content behind authentication instead.
Letting the file become unreachable
Under RFC 9309, a crawler that cannot reach robots.txt because of a server error (a 5xx status) must assume it should avoid the whole site. Monitor the file the way you monitor your home page.
Assuming one rule covers every AI company
Google-Extended applies to Google's AI use. Other companies publish their own tokens, and some crawlers may not follow the protocol at all.
Frequently asked questions
Will blocking Google-Extended hurt my search rankings?
Google says the token does not affect inclusion in Google Search and is not a ranking signal. It controls Gemini training and grounding use only.
Can robots.txt guarantee my content is never used for AI training?
No. It expresses a preference that compliant crawlers can honor. It cannot stop a crawler that ignores it, and it has no effect on content already collected.
Should I add Content-Usage rules now?
The syntax is still an Internet-Draft and can change. It is reasonable to follow its progress and wait for stable support from the crawlers you care about.
How often should I review my robots.txt?
Review it after site migrations, theme or plugin changes, and a few times a year, since crawler names and standards continue to evolve.
Key takeaways
- robots.txt states preferences for compliant crawlers; it is not access control.
- Google-Extended governs Gemini training and grounding use and does not affect Google Search inclusion or ranking.
- The Content-Usage examples (such as train-ai=n) come from an IETF draft. Separately, Google's documentation says it intends to support the AIPREF signal written as user-agent: __usage_ai-training in early 2027. Both are planned or in progress, not available as Google controls today.
- To keep a page out of search results, use noindex on a page that robots.txt does not block, and do not combine it with Disallow.
- Keep private content behind real access controls and add server-side protection for crawlers that ignore the rules.
Keep control of the site you publish
Deciding how automated visitors treat your content starts with owning the place where it lives. If you are setting up or moving a WordPress site, Z.com's WordPress Hosting page lists a web application firewall and AI bot protection among the security features on its plans. Review the plan details there to see whether they fit how you want to manage crawler traffic alongside your robots.txt choices.
Explore Z.com WordPress Hosting