Crawler readiness
AI crawler readiness checklist
A pre-launch checklist for public metadata, robots, sitemap, structured data, and llms.txt signals before you treat a site as crawler-ready.
Crawler readiness
Check public signals first
- Confirm the public homepage returns a successful fetch for the URL you intend to analyze.
- Review title, description, and OpenGraph-style metadata for the primary landing page.
- Check robots.txt for intentional allow/deny policy and a declared sitemap.
- Confirm /sitemap.xml or the declared sitemap exposes crawlable paths.
- Look for JSON-LD or other structured data types that match the page entity.
- Note whether llms.txt is present and whether its posture matches your machine-readable goals.
Crawler readiness
Do not overclaim
- Treat the readiness score as a diagnostic, not a ranking or traffic forecast.
- Only scan public sites you own, control, or have permission to inspect.
- Read topGaps evidence before chasing the overall score.
- Treat llms.txt as an emerging signal, not a universal crawler standard.
Crawler readiness
Named outputs to inspect
- score and maxScore give the overall diagnostic; categoryScores show where points were lost.
- topGaps[] names code, severity, evidence, and recommendation for each priority fix.
- remediation.share and remediation_checklist turn a snapshot into a shareable next-step list.
- before_after score_delta and gap code lists help compare a re-run after one visible patch.
- Use RapidAPI for marketplace auth, plans, and playground testing after the checklist.