# robots.txt for minecraftsearch.com # Default policy: allow every crawler full access. The explicit per-bot # groups below are intentionally redundant with the wildcard — they document # which crawlers we actively want and give us a place to add per-bot rules # later (crawl-delay, path hints, etc.). NOTE: a crawler obeys ONLY the most # specific User-agent group that matches it, so any future Disallow added to # "*" must also be added to the named groups below to take effect for them. User-agent: * Allow: / # ===== Search engines ===== # Googlebot ONLY, since 2026-09-22 (marketing/GOOGLE_RECOVERY_PLAN.md lever L1, # owner decision D2 in section 6.6.7). The four tool deep-link parameter families # below are ~46k "Crawled - currently not indexed" URLs that earn 4% of Google # impressions and 0.3% of its clicks, while Google has been in a serving demotion # since 2026-07-06. Blocking them costs nothing anywhere else: per the header # above, a crawler obeys ONLY its own most specific group, so Bingbot reads the # `Bingbot` group below and never sees these lines. That is the whole safety # argument, and Bing is ~95% of this site's organic traffic -- which is why # scripts/check-robots-txt.mjs pins BOTH halves: the four lines present here, and # every other group still a plain `Allow: /`. `Allow: /` stays below because # Google resolves a conflict by the LONGEST matching pattern, so `/*?item=` beats # `/` on a parameter URL while every other path still resolves to the Allow. User-agent: Googlebot Allow: / Disallow: /*?item= Disallow: /*?items= Disallow: /*?mob= Disallow: /*?seed= User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: Bingbot Allow: / User-agent: msnbot Allow: / User-agent: BingPreview Allow: / User-agent: DuckDuckBot Allow: / User-agent: Yandex Allow: / User-agent: YandexBot Allow: / User-agent: Baiduspider Allow: / User-agent: Applebot Allow: / # ===== AI training / answer engines ===== # Getting our content into AI answers (ChatGPT, Claude, Perplexity, etc.). # Note: Google-Extended and Applebot-Extended are training-control tokens, # not crawlers — Allow:/ here signals we permit AI training use of our content. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: Diffbot Allow: / User-agent: Amazonbot Allow: / Sitemap: https://minecraftsearch.com/sitemap.xml