# OpenAI Search and User-Triggered Fetches User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic Real-Time Fetches User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Perplexity AI Search User-agent: PerplexityBot Allow: / # You.com Search User-agent: YouBot Allow: / # OpenAI Training User-agent: GPTBot Disallow: / # Anthropic Training User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * # No blanket "Allow: /" here: Google picks the most specific rule, but order-based # parsers take the first match, and a leading Allow: / would swallow everything below. # Anything not matched by a Disallow is crawlable by default. # Article bodies are HTML fragments loaded into the blog modal by JS. They carry no # head, title or styling, so they must not be indexed as pages in their own right. Disallow: /blogs/ # Dated working copies of the live blog and home pages - near-duplicates, no canonical. Disallow: /blog15Apr.php Disallow: /blog18jun2026.php Disallow: /blog3sep2026.php Disallow: /home05Jun2026.php Disallow: /home19May2026.php Disallow: /home24Jun2026.php Disallow: /homev1.html Disallow: /home.html # Include files - these render as broken partial pages if requested directly. Disallow: /header.php Disallow: /headerv1.php Disallow: /footer.php Disallow: /footerv1.php Disallow: /canonical.php # Sitemap Location Sitemap: https://www.werafoods.com/sitemap.xml