Search and AI Optimizer technical details

What a scan reads

1. robots.txt at the root of the host you enter, read up to 500 KiB, the limit Google applies.
2. The page: the homepage, or the exact URL you enter. Only its <head> matters here: the canonical link, robots meta tags and <link rel="alternate"> feed links, plus the X-Robots-Tag header.
3. llms.txt at the root, if there is one.
4. Host variants: http:// and the www or non-www twin, to see whether they redirect to one address.
5. Feeds linked from the page. If none are linked, a few common paths are tried (/feed, /feed.xml, /rss.xml, /atom.xml, /index.xml) and the first feed found is used.
6. Sitemaps listed in robots.txt. If robots.txt lists none, /sitemap.xml and /sitemap_index.xml are tried. Sitemap indexes are followed breadth first, and gzipped sitemaps are decompressed.
7. A sample of listed URLs: the URL you entered, the newest feed items, up to three llms.txt links, and URLs spread evenly through each sitemap (plus its newest entries), not just the first few.

Limits

When a limit is reached, the results say so under the scan coverage, for example “Read 10 of 46 sitemaps”. Findings then describe only what was read.

Safe fetching

• Only http and https on the standard ports are fetched, never addresses with a user name or password, and never IP addresses typed directly.
• Before every request, including each redirect, the host name is looked up over DNS-over-HTTPS. If any address it returns is private, loopback, link-local, carrier-grade NAT or otherwise not public, the request is not made.
• Steps after the first only fetch the scanned site (its host, the www or non-www twin and their subdomains) and hosts its robots.txt names in Sitemap lines. A signed scan token, valid for 15 minutes, carries that list.
• Requests use the user agent DNS-Panopticon-SearchAI/1.0. The scanner never pretends to be Googlebot or an AI crawler.
• Nothing is stored: results exist only in your browser.

How robots.txt is read

Rules are applied as RFC 9309 describes. A crawler uses every group that names its token, merged; if none does, it uses the * groups. Within those rules the longest matching path wins, and Allow wins a tie. * matches any characters and a trailing $ anchors the end. /robots.txt itself is always allowed.

• No robots.txt (4xx): everything is allowed.
• Server error or no answer (5xx, 429, timeout): the whole site is treated as blocked, because that is what the standard tells crawlers to assume.
• Crawl-delay, Content-Signal and fields such as Noindex or Host are reported, but they don’t change access. Content-Signal lines state how you want content used; crawlers are not required to follow them.

Crawler purposes

The crawler access matrix groups each agent by what it is for. The catalog follows each vendor’s own documentation.

Whether to allow training crawlers is your choice, so it never lowers a rating. The report only notes it, along with an unusual mix such as allowing a vendor’s training crawler while blocking its search crawler.

Simulated, not observed

The matrix applies your robots.txt and tags to each crawler. It is not a log of crawler visits. Firewalls, bot protection and IP rules can treat real crawlers differently from the way your robots.txt reads, and some user-initiated fetchers may not consult robots.txt at all. To confirm real activity, check your server logs and each provider’s tools, such as Search Console’s crawl stats.

Checks

• robots.txt: reachability, content type, size, lines crawlers can’t parse, unsupported fields, Sitemap lines, and whether major search or AI search crawlers are blocked.
• Sitemaps: well-formed XML, the sitemaps.org namespace, the 50,000 URL and 50 MB limits, nested indexes, relative or duplicate URLs, URLs on other hosts, http URLs on an https site, and lastmod dates that are malformed, in the future, or all the same.
• Feeds: RSS 2.0, RSS 1.0 or Atom; required elements; readable dates; items without links; whether the page links to the feed; and how old the newest item is.
• Checked URLs: errors, redirects, noindex in a meta tag or X-Robots-Tag, a canonical pointing elsewhere, and nosnippet or max-snippet:0, which keep a page out of quoted answers.
• Hosts: http:// that doesn’t redirect to https://, and www and non-www both serving the site.
• llms.txt: if present, its title, summary and links. It is an experimental proposal that major search and AI providers have not said they use, so a missing llms.txt is only a note.

Contradictory signals

The most useful findings compare files. A sitemap URL that robots.txt blocks, that is marked noindex, that redirects or that names another canonical URL asks crawlers for two opposite things. So does a sitemap that robots.txt blocks, sitemap URLs on a different host from the one the site settles on, feed items missing from the sitemap, and llms.txt links to pages blocked for AI search.

Ratings

Three ratings summarize the findings: Search, Answer engines and Generative engines. Each is Good, Needs attention or Problems, depending on whether any finding that affects it is a warning or a problem. There is no score, because no file guarantees indexing, ranking or citations in AI answers.

Submission hub

Search engines accept sitemap submissions: Google Search Console, Bing Webmaster Tools, Yandex Webmaster, Naver Search Advisor and Baidu’s Search Resource Platform. IndexNow notifies Bing, Yandex, Naver, Seznam and other participating engines when URLs change; Google does not take part. AI providers have no submission: for them robots.txt is the whole setup. Google retired its sitemap ping endpoint in 2023, so a robots.txt Sitemap line or a Search Console submission is how Google learns of a sitemap.