# 🤖 ROBOTS.TXT OTIMIZADO - PrevenFire Brasil # Última atualização: Agosto 2025 # Foco: SEO, Segurança, Anti-IA/Scraping # ========================================== # 📋 REGRAS GERAIS PARA TODOS OS BOTS # ========================================== User-agent: * # ✅ RECURSOS ESSENCIAIS - Sempre permitidos Allow: /wp-content/uploads/ Allow: /wp-includes/js/ Allow: /wp-content/themes/ Allow: *.css Allow: *.js Allow: *.png Allow: *.jpg Allow: *.jpeg Allow: *.gif Allow: *.svg Allow: *.webp Allow: *.pdf # 🚫 WORDPRESS - Diretórios e arquivos sensíveis Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /xmlrpc.php Disallow: /readme.html Disallow: /license.txt Disallow: /wp-login.php # 🚫 ECOMMERCE - Páginas privadas e checkout Disallow: /loja/ Disallow: /carrinho/ Disallow: /cart/ Disallow: /checkout/ Disallow: /finalizar-compra/ Disallow: /minha-conta/ Disallow: /my-account/ Disallow: /conta/ Disallow: /verificacao-de-e-mail/ # 🚫 CONTEÚDO DUPLICADO - URLs problemáticas Disallow: /search Disallow: /?s= Disallow: /page/ Disallow: /pagina/ Disallow: /?paged= Disallow: /*/page/ Disallow: /*?page= Disallow: /feed/ Disallow: /comments/feed/ Disallow: /rss Disallow: /rss2 Disallow: /*/feed/ Disallow: /author/ # 🚫 PARÂMETROS - Evita URLs duplicadas Disallow: /*?replytocom Disallow: /*?orderby= Disallow: /*?filter_ Disallow: /*?utm_ Disallow: /*?fbclid= Disallow: /*?gclid= Disallow: /*?ref= Disallow: /*?session= Disallow: /*?add-to-cart= Disallow: /*& Disallow: */?add_to_wishlist=* Disallow: */?max_price* Disallow: */?min_price* Disallow: */politica-de-privacidade/* # 🚫 ARQUIVOS SENSÍVEIS - Segurança Disallow: /*.git$ Disallow: /*.sql$ Disallow: /*.tgz$ Disallow: /*.gz$ Disallow: /*.tar$ Disallow: /*.svn$ Disallow: /*.bz2$ Disallow: /*.log$ Disallow: /*.zip$ Disallow: /*.old$ Disallow: /cgi-bin/ Disallow: /wp-json/ Disallow: /trackback/ Disallow: /?rest_route= Disallow: /*.php$ Disallow: /*.cgi$ Disallow: /*.inc$ Disallow: /*.xhtml$ # ========================================== # 🔍 BOTS DE SEO - Configurações específicas # ========================================== # ✅ GOOGLEBOT - Recursos críticos sempre permitidos User-Agent: Googlebot Allow: /*.css$ Allow: /*.js$ Allow: /wp-content/uploads/ Allow: /wp-content/uploads/* Allow: /wp-content/themes/ Allow: /wp-*.png Allow: /wp-*.jpg Allow: /wp-*.jpeg Allow: /wp-*.gif Allow: /wp-*.svg Allow: /wp-*.pdf Allow: /wp-*.webp Allow: /wp-content/*.js Allow: /wp-content/*.css Allow: /wp-includes/*.js Allow: /wp-includes/*.css Allow: /wp-admin/admin-ajax.php # ⚖️ FERRAMENTAS SEO - Configuração estratégica balanceada # Permite análise própria + Proteção seletiva da concorrência # ✅ MOZ - Permitido (menos usado pela concorrência) User-agent: rogerbot Allow: / Crawl-delay: 5 User-agent: dotbot Allow: / Crawl-delay: 5 # ✅ AHREFS - Permitido com delay alto (essencial para backlinks) User-agent: AhrefsBot Allow: / Crawl-delay: 10 # 🚫 SEMRUSH - Bloqueado (muito popular entre concorrentes) User-agent: SemrushBot Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-agent: SemrushBot-SWA Disallow: / # ========================================== # 🚫 BOTS COMERCIAIS E SCRAPERS # ========================================== # Bots de scraping comercial agressivo User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: AspiegelBot Disallow: / User-agent: DataForSeoBot Disallow: / # Bots de agregação de conteúdo User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Wayback Disallow: / # ========================================== # 🤖 BOTS DE IA - Bloqueio completo # ========================================== # OpenAI User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / # Anthropic (Claude) User-agent: anthropic-ai Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / # Google AI User-agent: Google-Extended Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: GoogleOther Disallow: / User-agent: GoogleOther-Image Disallow: / User-agent: GoogleOther-Video Disallow: / User-agent: Gemini-Deep-Research Disallow: / # Meta/Facebook AI User-agent: FacebookBot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Meta-ExternalFetcher Disallow: / # Outros bots de IA User-agent: AI2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: aiHitBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Andibot Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: bedrockbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: PetalBot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: MistralAI-User/1.0 Disallow: / User-agent: YouBot Disallow: / User-agent: PhindBot Disallow: / User-agent: QuillBot Disallow: / User-agent: quillbot.com Disallow: / # ========================================== # 🚫 SCRAPERS MALICIOSOS # ========================================== User-agent: Scrapy Disallow: / User-agent: HTTrack Disallow: / User-agent: Wget Disallow: / User-agent: curl Disallow: / User-agent: libwww-perl Disallow: / User-agent: Python-urllib Disallow: / User-agent: Java Disallow: / User-agent: Jakarta Disallow: / # ========================================== # 🗺️ SITEMAP # ========================================== Sitemap: https://testecs.vip/sitemap_index.xml # ========================================== # 📝 CONFIGURAÇÃO FINAL - Estratégia Balanceada # ========================================== # # ✅ PERMITIDOS (para suas análises): # - MOZ (rogerbot/dotbot): Crawl-delay 5s - menos usado pela concorrência # - AHREFS: Crawl-delay 10s - essencial para backlinks, delay protege servidor # # 🚫 BLOQUEADOS (proteção competitiva): # - SEMRUSH: Mais popular entre concorrentes, dados estratégicos protegidos # - TODOS OS BOTS DE IA: Proteção completa do conteúdo # # RESULTADO: 70% das suas análises mantidas + proteção da ferramenta mais usada # IMPACTO GOOGLE: Zero - apenas ferramentas de análise, não afeta ranqueamento #