# LocalAI Recipes — robots.txt
# Agent-first: fetchers that identify honestly are welcome. Content is
# CC-BY-4.0 with attribution; machine reuse is the point.

# ---- Fair-use rate limit (kanban t_d6be9581) ----
# The website's HTML pages sit behind a SOFT per-client-IP fair-use gate:
# extreme sustained rates (over 120 requests/minute, short bursts fine)
# get HTTP 429 + Retry-After with a self-describing body naming the limit
# and this policy. NOT a ban, not a captcha. /api/* and /healthz are
# EXEMPT — API traffic is what this site is for. Enforcing since
# 2026-10-13 (an incident-free observe week preceded the flip; the owner
# steer was soft fair-use only — no bans, no captcha). Full semantics:
# /llms.txt, "Website lane rate limits".

# ---- AI training / data-mining preferences (AIPREF-style) ----
# Aggregating recipe text into AI training corpora: allowed with attribution.
User-agent: *
Allow: /
# Modest crawl-delay for unannounced crawlers. Named, well-behaved bots
# below self-manage their rate and carry no crawl-delay; the fair-use
# gate above catches any that go extreme anyway.
Crawl-delay: 1

# ---- LLM / agent crawlers: explicitly welcome ----
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Disallow: / # refusal-to-attribute reuse is not in the spirit of the license

# ---- Search / misc ----
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# ---- Never crawl: auth + write surfaces ----
# (harmless: they 401/405, but be polite)
Disallow: /api/v1/keys
Disallow: /keys/
Disallow: /admin

Sitemap: https://localairecipes.com/sitemap.xml