For site owners
PlainTermsBot
PlainTermsBot is the crawler behind PlainTerms. It reads the public legal documents a website asks its visitors to accept — terms of service, privacy policy, cookie policy — so PlainTerms can grade them and quote the clauses that matter.
How to recognize it
Every request it makes carries this user agent:
PlainTermsBot/1.0 (+https://api.plain-terms.app/bot)What it fetches
- Your home page,
/robots.txt, and your sitemap, to find where your policies are published. - Pages that look like legal documents: links labelled terms, privacy, cookies, and the like, plus a short list of conventional paths such as
/termsand/privacy. - Links to your legal documents that a visitor’s browser found on one of your pages. When a PlainTerms user checks a site nobody has analyzed yet, the extension can send up to 10 of them — useful when your footer is built by JavaScript. Only links on your own domain are accepted, with any query string removed, and they are fetched last, after everything above, and checked like anything else.
- Pages on other domains only when your site links to them as a legal document, such as terms hosted by a parent company.
It does not log in, send cookies, submit forms, or download anything larger than 5 MB. It follows up to 5 redirects in a row.
When it visits
When a PlainTerms user checks a site that has not been analyzed yet, and later to see whether a document has changed. Each service is analyzed once and the result is shared with every PlainTerms user, so your site is not fetched once per visitor. Repeat visits send If-None-Match and If-Modified-Since, so an unchanged page usually costs a 304.
How much it fetches
- At most 25 pages per visit, plus one robots.txt for each host it reads from, and never more than 2 links away from your home page.
- At most 4 connections to your server at a time, at least 250 ms apart — redirects included, and longer if your robots.txt sets a
Crawl-delay. - Every visit stops after 60 seconds, with whatever it has found, and sooner once it has your terms, privacy policy, and cookie policy.
robots.txt
PlainTermsBot reads the robots.txt of every host it fetches from, and follows the group addressed to User-agent: PlainTermsBot if there is one, otherwise the * group. There is one documented exception: paths that name a legal document. A site cannot both ask visitors to agree to its terms and forbid reading them, so these paths are fetched even when disallowed — still throttled, and still identified:
/terms /tos /legal /privacy /policies /policy /cookie /cookies /acceptable-use /eula /user-agreement
The exception also covers paths beneath them, and paths that continue them after a hyphen, such as /legal/privacy and /terms-of-service. To keep PlainTermsBot out of everything else on your site:
User-agent: PlainTermsBot
Disallow: /A missing robots.txt, or one answered with a 403 or another 4xx, means no restrictions, as RFC 9309 specifies. One that fails with a server error or does not answer means the whole site is treated as disallowed, and only the paths above are read.
If you block it
PlainTermsBot does not try to get around a refusal. A bot challenge, a 429, or a 451 ends the visit, and so does a 403 on any page it did not guess at. (On the conventional paths above, plenty of servers answer a page that does not exist with a 403, so there it counts as a 404.) After a refusal it waits at least a day before looking for your policies again, and a site that refuses twice in a row is marked as blocked.