Your next big traffic source is a crawler you might be blocking.

Most websites accidentally block AI crawlers in robots.txt — or ship pages they literally cannot read. Avuril finds every gap, scores your readiness out of 100, and watches it around the clock.

avuril.com/sites/example.comformula v2

82

Readiness score

Grade B

  • GPTBot Allowed
  • OAI-SearchBot Allowed
  • ClaudeBot Blocked
  • PerplexityBot Allowed
  • Google-Extended Blocked

Example report — every verdict backed by reproducible HTTP evidence.

From shops, restaurants, and B2B teams

  • ChatGPT couldn't see our products. Avuril found a leftover robots.txt blocking GPTBot. AI shopping referrals are up 34%.
    Shopify Store Owner
  • Our menu was a blank page to crawlers, so ChatGPT sent people next door. Avuril showed the bot view. Reservations from AI answers are up 27%.
    Manhattan Restaurant Owner
  • Claude was citing a competitor because our docs were noindex. Avuril caught the staging header. Qualified inbound is up 41%.
    B2B SaaS Owner
  • A Shopify app started 403-ing SearchBot and collection traffic went quiet. Avuril caught it on the next scan. Sessions from AI search are up 22%.
    Shopify Store Owner
  • Perplexity had our hours wrong and no specials. Avuril found we were blocking the crawler. Weekend covers are up 19%.
    Brooklyn Restaurant Owner
  • Prospect copilots couldn't read our product pages. Avuril unblocked Claude and Perplexity. Demo requests are up 29%.
    B2B Logistics Owner
  • A plugin rewrite blocked GPTBot overnight. Avuril emailed me before morning. We recovered 18% of the AI-driven checkout traffic we'd lost.
    Shopify Store Owner
  • An old robots.txt blocked GPTBot and Google-Extended. One Avuril scan fixed it. Direct booking traffic is up 31%.
    Austin Restaurant Owner
  • SearchBot was stuck behind a WAF rule we didn't know about. Avuril flagged the 403. Docs-sourced pipeline is up 2.1x.
    B2B HR Tech Owner

Deterministic checks. No AI guessing.

Every finding is reproducible HTTP evidence — not a model's opinion.

15 AI crawlers, one verdict each

Exact robots.txt rules evaluated per bot — search, agent, and training classes. Spot accidental blocks of OAI-SearchBot or PerplexityBot in seconds.

Bot-view parity

We fetch your pages twice: as a browser and with our verifier probing as GPTBot, ClaudeBot, or PerplexityBot. If the crawler-shaped request gets a 403 or an empty JavaScript shell, you'll know.

llms.txt validation & generation

Adoption grew 8.8x in a year. Validate your file against the spec, or copy a starter llms.txt that maps your homepage for AI systems and coding agents.

Continuous monitoring

Paid plans re-scan daily (Free weekly). When a deploy breaks AI visibility, the next sweep raises an in-app alert — and email on Solo and Agency.

Regression alerts

A new block on a search crawler, a dropped score, a stray noindex on your homepage — alerts land in-app for every plan, and in email on Solo and Agency.

Zero install

No scripts, no middleware, no DNS changes. Paste a domain; we fetch public URLs with our verifier user-agent.

Surface security, no pentest theater

HSTS, CSP, cookie flags, and accidental public .env/.git — from the same HTTP fetch. No AI APIs, no paid scanners.

What attacking agents take

Keys in public JS, open JSON collections, hidden instructions for visiting copilots. The first pass a recon agent runs — before you hear about it from someone else.

Three steps, under five minutes

  1. 1.

    Paste a domain

    Any public URL. Check the five biggest crawlers free with no account, or sign up for the full report.

  2. 2.

    Get the report

    A deterministic 0–100 readiness score with per-bot verdicts, prioritized fixes, and the evidence behind every finding.

  3. 3.

    Fix and monitor

    Apply the fixes (most take minutes), then let scheduled re-scans guard against regressions.

Stop guessing what AI sees.

Know which crawlers can read your site, and get alerted after the next scheduled sweep. Free tier included.