Crawler readiness

AI crawler readiness API for public websites

Check whether a public website exposes crawler-readable metadata, sitemap, robots, structured data, and llms.txt signals.

View API details Open in RapidAPI

Problem

Crawler and machine-readable site signals are easy to miss before launch.

Public sites often ship with partial metadata, missing structured data, unclear sitemap discovery, or an unreviewed llms.txt posture. A shallow diagnostic gives teams a repeatable way to inspect public signals, prioritize gaps, patch one visible signal, and re-run the check.

When to use it

  • You want a pre-launch check for crawler-readable metadata and policy signals.
  • You need an evidence-backed checklist for structured data, sitemap, robots, and llms.txt readiness.
  • You want a stored snapshot to become a shareable remediation checklist.
  • You want to compare readiness scores before and after patching public site signals.

First call

Run a small request and inspect the response.

curl -X POST 'https://website-intelligence-api.linkridge.net/v1/site-intel/analyze' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: <your-api-key>' \
  -d '{
  "url": "https://linkridge.net/"
}'

Use RapidAPI for marketplace auth, plans, and interactive testing.

Example response

{
  "domain": "linkridge.net",
  "requestedUrl": "https://linkridge.net/",
  "fetchability": {
    "ok": true,
    "status": 200
  },
  "score": 50,
  "maxScore": 100,
  "categoryScores": {
    "fetchability": { "score": 20, "max": 20 },
    "metadata": { "score": 12, "max": 25 },
    "crawlPolicy": { "score": 8, "max": 20 },
    "structuredData": { "score": 0, "max": 10 },
    "llmReadiness": { "score": 10, "max": 15 },
    "freshness": { "score": 0, "max": 10 }
  },
  "topGaps": [
    {
      "code": "missing_structured_data",
      "severity": "medium",
      "evidence": "No JSON-LD structured data types were extracted.",
      "recommendation": "Add schema.org JSON-LD for the relevant entity types."
    },
    {
      "code": "missing_sitemap_signal",
      "severity": "medium",
      "evidence": "No sitemap was declared in robots.txt and /sitemap.xml did not expose a sitemap.",
      "recommendation": "Add a sitemap and reference it from robots.txt."
    }
  ],
  "remediation": {
    "path": "/v1/site-intel/domains/linkridge.net/remediation",
    "share": {
      "title": "Website Intelligence remediation checklist for linkridge.net",
      "summary": "Score up by 14 points; 2 open remediation items."
    },
    "before_after": {
      "score_delta": 14,
      "resolved_gap_codes": ["missing_llms_txt"],
      "new_gap_codes": ["missing_open_graph_summary"],
      "persistent_gap_codes": ["missing_structured_data"]
    },
    "remediation_checklist": [
      {
        "status": "open",
        "code": "missing_structured_data",
        "first_seen": "persistent"
      }
    ]
  }
}

Based on