robots.txt & llms.txt Checker API

robots.txt + llms.txt checker: crawler access verdicts for major search AND AI bots (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and more), declared sitemaps, syntax warnings, and llms.txt / llms-full.txt presence with structure summary. ?url=any page on the site

Parameters

Example request

curl "https://api.webbersites.com/api/seo/robots-check?url=https%3A%2F%2Fexample.com"
# first call returns 402 + payment requirements; an x402 client pays and retries automatically

Example response

{
    "origin": "https://example.com",
    "robots": {
        "found": true,
        "bots": {
            "GPTBot": {
                "kind": "ai",
                "root_blocked": true
            },
            "Googlebot": {
                "kind": "search",
                "root_blocked": false
            }
        },
        "sitemaps_declared": [
            "https://example.com/sitemap.xml"
        ]
    },
    "llms_txt": {
        "found": false
    }
}

How payment works

There is no signup and no API key. Call the endpoint; it replies 402 Payment Required with machine-readable payment requirements. Your client signs a USDC transfer authorization (EIP-3009, gasless) and retries with the X-PAYMENT header — @x402/fetch does this automatically. See the overview for a working snippet.