# Urnovl robots.txt # Last updated: 2025 # =================== # AdMob app-ads.txt Verification # =================== User-agent: Google-adstxt Disallow: User-agent: Mediapartners-Google Disallow: User-agent: Googlebot Disallow: # =================== # Default (All Crawlers) # =================== User-agent: * Allow: / # Creation/Edit forms Disallow: /novl/create Disallow: /page/create Disallow: /competitions/create # Private user profile routes Disallow: /subscriptions Disallow: /user/*/profile/settings Disallow: /user/*/profile/earnings Disallow: /user/*/profile/transactions Disallow: /user/*/profile/stats Disallow: /user/*/profile/partnership Disallow: /user/*/profile/pages Disallow: /user/*/profile/following-pages # System routes Disallow: /account/deletion-request/result Disallow: /app # =================== # AI Training Crawlers # =================== # Allow: marketing, listings, public profiles # Disallow: user-generated content (stories) User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: ClaudeBot User-agent: Google-Extended User-agent: Bytespider User-agent: Amazonbot User-agent: FacebookBot User-agent: PerplexityBot User-agent: YouBot User-agent: Applebot-Extended # Landing & Marketing Allow: /$ Allow: /who-is-it-for/ Allow: /features/ Allow: /for-publishers # Listings (without query params) Allow: /home$ Allow: /stories$ Allow: /pages$ Allow: /browse$ Allow: /competitions$ # Block story summary modals Disallow: /*?novl= Disallow: /*?page= # Public profiles & pages Allow: /user/ Allow: /page/ # Legal & Info Allow: /faqs Allow: /privacy-policy Allow: /terms-of-service Allow: /terms-of-pay Allow: /house-rules Allow: /partnership-program # Block story content Disallow: /novl/ Disallow: /competitions/create Disallow: / # Crawl rate limit Crawl-delay: 2 Sitemap: https://www.urnovl.co/sitemap.xml