Crawl · NinjaOps Guides

robots.txt for Affiliate Sites: Declarations and Mistakes

robots.txt is your crawl instruction manual. The classic mistakes: blocking CSS/JS, forgetting the Sitemap line, and Disallow-ing pages you actually want ranked.

How to fix it

  1. Keep robots.txt at the domain root, reachable at 200.
  2. Add one Sitemap: line with the absolute URL.
  3. Don't block assets — blocked JS breaks rendering.
  4. Use Disallow only for parameters, internal search, staging.
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml

Diagnose this exact issue on your site — free

The 100-Point Affiliate Site Grader runs this check (and 40+ more) on your live site in ~10 seconds. No signup.

Want it fixed for you?

The Zero-Overhead Affiliate Playbook walks through every gap in this guide in 14 days on a $0 stack. Join the launch list →

More Crawl fixes: