# ============================================================================ # ROBOTS.TXT — Florida Living Outdoor # https://floridalivingoutdoor.com/robots.txt # ============================================================================ # Prepared by: Cerberus | The Pantheon | Oculus Intel # Revised by: Archer | Oculus Intel # Version: 2.0 # Date: October 2026 (v1.0 February 2026) # AI Policy: MAXIMIZE VISIBILITY — All AI crawlers allowed # ============================================================================ # # HOW THIS FILE WORKS # A crawler obeys only the group that names it most specifically. Bot-specific # groups do NOT inherit the "User-agent: *" rules. To keep every bot named AND # make every rule apply to all of them, all user-agents below share ONE group: # the User-agent lines in Layers 1–3 are followed by a single rule set. # Do not insert rule lines or blank-line-separated groups between them. # ============================================================================ # ───────────────────────────────────────────────────────────────────────────── # LAYER 1: UNIVERSAL CRAWLERS # Default rules for all compliant search engine and web crawlers. # This is the baseline access policy. # ───────────────────────────────────────────────────────────────────────────── User-agent: * # ───────────────────────────────────────────────────────────────────────────── # LAYER 2: MAJOR SEARCH ENGINE SPECIFICS # Primary search engines, named explicitly. They share the rules below. # ───────────────────────────────────────────────────────────────────────────── # Google (primary search engine — full access) User-agent: Googlebot # Google Images User-agent: Googlebot-Image # Bing (search) and Microsoft Copilot User-agent: Bingbot # DuckDuckGo User-agent: DuckDuckBot # Yahoo / Bing partnership User-agent: Slurp # Yandex (Russian search engine — allow for international visibility) User-agent: Yandex # Apple (Siri, Spotlight, Apple Maps) User-agent: Applebot # ───────────────────────────────────────────────────────────────────────────── # LAYER 3: AI CRAWLER POLICY # Client directive: MAXIMIZE AI VISIBILITY # All AI training and inference crawlers are ALLOWED. # This ensures Florida Living Outdoor content can appear in: # - ChatGPT / OpenAI responses # - Google Gemini # - Claude / Anthropic responses # - Perplexity AI answers # - Meta AI responses # - Other AI-powered search and answer engines # Note: Google AI Overviews and AI Mode are controlled in Search Console # (Settings → Search generative AI = Include), not by robots.txt. # ───────────────────────────────────────────────────────────────────────────── # OpenAI (ChatGPT, SearchGPT) User-agent: GPTBot # OpenAI user-initiated browsing User-agent: ChatGPT-User # OpenAI search index (ChatGPT search results) User-agent: OAI-SearchBot # Google (Gemini and AI training use) User-agent: Google-Extended # Anthropic (Claude) User-agent: ClaudeBot # Anthropic search index and user-requested fetches User-agent: Claude-SearchBot User-agent: Claude-User # Anthropic (legacy user agent) User-agent: anthropic-ai # Perplexity AI User-agent: PerplexityBot # Perplexity user-requested fetches User-agent: Perplexity-User # Meta AI User-agent: FacebookBot User-agent: Meta-ExternalAgent # Common Crawl (open dataset used by many AI systems) User-agent: CCBot # Cohere AI User-agent: cohere-ai # Amazon / Alexa User-agent: Amazonbot # Apple AI / Siri intelligence User-agent: Applebot-Extended # Bytedance / TikTok AI User-agent: Bytespider # ───────────────────────────────────────────────────────────────────────────── # SHARED RULES — apply to every user-agent listed above # ───────────────────────────────────────────────────────────────────────────── # Allow: Critical resources for proper page rendering Allow: / # Disallow: Admin and system directories Disallow: /admin/ Disallow: /admin-* Disallow: /_/ Disallow: /_next/ Disallow: /api/ # Disallow: Staging and development environments Disallow: /staging/ Disallow: /dev/ Disallow: /test/ # Disallow: Internal search results (if applicable) Disallow: /search? Disallow: /*?q= Disallow: /*?s= # Disallow: Thank-you / confirmation pages (low-value, no-index content) Disallow: /thank-you Disallow: /thank-you/ # Disallow: Shopping cart, checkout, account pages (if e-commerce exists) Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /login/ Disallow: /signup/ # Disallow: Duplicate content paths and URL fragments # NOTE: The site has blog posts accessible via multiple URL patterns # (/b/, /post/, /c/.../b/). Canonical tags on each page are the # primary solution for duplicate content. The robots.txt does NOT # block these paths because all versions should remain crawlable # for Google to discover and respect the canonical tag. # Archer rule: blocking duplicate paths comes only after the blog archive # page is live and verified, with owner approval. # Disallow: Utility and system files Disallow: /cdn-cgi/ Disallow: /*.json$ Disallow: /wp-admin/ # ───────────────────────────────────────────────────────────────────────────── # LAYER 4: SITEMAP DECLARATIONS # Reference to XML sitemaps for search engine discovery. # ───────────────────────────────────────────────────────────────────────────── # NOTE: Verify the exact sitemap URL(s) in the site's CMS or hosting platform. # Update the paths below to match the actual sitemap location. Sitemap: https://floridalivingoutdoor.com/sitemap.xml # Sitemap: https://floridalivingoutdoor.com/sitemap-posts.xml # Sitemap: https://floridalivingoutdoor.com/sitemap-pages.xml # ───────────────────────────────────────────────────────────────────────────── # CRAWL RATE # No explicit Crawl-delay is set. Google ignores Crawl-delay directives. # If the site experiences crawl budget issues, address via Google Search # Console's crawl rate settings rather than robots.txt directives. # ───────────────────────────────────────────────────────────────────────────── # ============================================================================ # END OF ROBOTS.TXT # ============================================================================