Search & discovery
Search and AI Optimizer
Enter a domain or a page URL. We read its robots.txt, sitemaps, feeds and llms.txt, check a sample of the pages they list, and show how well the site supports search engines, answer engines and AI assistants, with what to fix first.
What this checks
- robots.txtCrawler rulesRead the way RFC 9309 says crawlers should read it, for each major search and AI agent.
- SitemapsSitemaps and indexesValidity, limits, lastmod dates and URLs robots.txt blocks.
- FeedsRSS and AtomRequired fields, dates and how recently the site published.
- PagesListed URLsErrors, redirects, noindex and canonical tags on a sample.
Allowing a crawler or submitting a sitemap helps search and AI tools find a site. It does not guarantee indexing, ranking or citations in AI answers. llms.txt is reported as an optional, experimental file.