public/robots.txt

   1 # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
   2 #
   3 # To ban all spiders from the entire site uncomment the next two lines:
   4 # User-agent: *
   5 # Disallow: /