Every request we make identifies itself. Control-file fetches (robots.txt, llms.txt, sitemaps) send:
Mozilla/5.0 (compatible; AvurilVerifier/1.0; +https://avuril.com/bots)
Parity probes append a label naming the crawler whose access rules we are testing — the identity stays ours:
Mozilla/5.0 (compatible; AvurilVerifier/1.0; +https://avuril.com/bots) probing-as/GPTBot
This page explains what the fetcher does and how to control it.
We fetch /robots.txt and evaluate which rules apply to well-known AI crawler user-agents. We do not need to crawl your pages for this.
When you request a scan, we fetch a small number of public URLs twice: once with a standard browser user-agent, and once with our verifier user-agent carrying a probing-as label for the crawler being tested. We never impersonate GPTBot, ClaudeBot, or any other operator — every request is attributable to us.
Fetches only happen when you (or a scheduled re-scan you own) request one. We never crawl at search-engine scale and respect standard caching etiquette.
On a full scan we also fetch a short allowlist of well-known public paths (for example /.env, /.git/HEAD, /.well-known/security.txt) with a tiny byte cap. We record whether the path matched a secret-file fingerprint — never the file body. The unauthenticated GPTBot checker does not run these probes.
Signed-in scans also look at public JS for credential prefixes (never the full secret), GET a handful of unauthenticated JSON collection paths, OPTIONS the origin for CORS *, and read AGENTS.md / mcp.json if they are published. Hidden HTML that looks like an instruction to a visiting agent is flagged from the homepage body. We do not send exploit payloads or iterate other users' IDs.
The verifier honors robots.txt directives that apply to its user-agent token AvurilVerifier (or the wildcard group). To stop our page fetches, add the following to your robots.txt:
User-agent: AvurilVerifier Disallow: /
With that rule in place we skip bot-view parity, machine-hygiene, surface-security, and agent-surface checks entirely, and the report says so. We still read /robots.txt and /llms.txt — those are the control files every crawler reads, and reading them is the only way to report what they say.