Which pages does the audit crawl?
Up to 6 pages per audit: your homepage plus the top 5 internal pages.
Last updated: 2026-05-05
Which pages does the audit crawl?
The audit doesn't crawl your entire site — that would take hours and over-weight low-value URLs. Instead it samples a representative set chosen to reflect how AI models actually evaluate a domain.
The sampling logic
For each audit run, Pranas selects:
- The homepage (always)
- Top-level navigation pages discovered from the homepage's main nav — typically Product, Features, Pricing, About
- The blog index if present, plus a sample of recent articles
- The sitemap's highest-priority URLs if a valid
sitemap.xmlexists - Pages flagged in your
llms.txtif present
The total sample is capped at roughly 15–25 pages to stay within rate-limit and credit budgets while still being representative.
Per-page fetch strategy
Each sampled page is fetched twice:
- Once via Firecrawl with Google's user agent
- Once via Firecrawl with Bing's user agent
If either fetch is blocked or returns an error, Pranas falls back to a generic browser user agent so the audit still completes — but the block itself is logged as a Discoverability finding.
The dual-fetch catches the surprisingly common case where a site serves different markup, schema, or even different content to different crawlers (often unintentionally, via CDN rules or a stale WAF policy).
Sampling reference
| Page group | Always included? | How it is discovered |
|---|---|---|
| Homepage | Yes | Directly from the domain you enter |
| Top-level navigation pages | Usually | Main nav links on the homepage |
| Blog index and recent articles | If present | Homepage links and sitemap |
| Highest-priority sitemap URLs | If a valid sitemap.xml exists | sitemap.xml priority values |
| Pages listed in llms.txt | If llms.txt exists | Root-level llms.txt |
| Authenticated pages | No | Excluded — the audit is unauthenticated |
| robots.txt-disallowed paths | No | Excluded and logged as a finding |
| Non-HTML files (PDF, images, JSON) | No | Excluded |
The total sample is capped at roughly 15–25 pages so a run completes in 30–90 seconds.
Why not crawl everything?
- Diminishing returns. Issues found on 10 representative pages are usually present site-wide. Auditing 1,000 pages confirms what 10 already told you.
- Rate limits. Firecrawl, Schema parsers, and downstream scorers all have quotas. Sampling keeps the audit completable in 30–90 seconds.
- Actionability. A 200-page issue list is overwhelming. A focused list across 20 representative pages is fixable in a sprint.
Customising what's audited
Currently the sampling is automatic. To bias the audit toward specific pages:
- Add them to your
llms.txt - Ensure they're in your
sitemap.xmlwith a highpriorityvalue - Link to them from your homepage's main navigation
Per-URL audit selection is on the roadmap.
Pages that are excluded
- Authenticated / behind-login pages (the audit is unauthenticated)
- Pages explicitly disallowed in your
robots.txt - Non-HTML resources (PDFs, images, raw JSON endpoints)
Related articles
- AI Search Audit (GEO Audit)
- Top Fixes per pillar
- Audit rate limit