From: Neil Smith Date: Wed, 8 Jan 2014 14:17:16 +0000 (+0000) Subject: robots.txt now bans all robots X-Git-Url: https://git.njae.me.uk/?p=feedcatcher.git;a=commitdiff_plain;h=7f3da99f73d86dacf7e79422a8af1db4e5261abe robots.txt now bans all robots --- diff --git a/public/robots.txt b/public/robots.txt index 1a3a5e4..b19a377 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,5 +1,5 @@ # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: -# User-agent: * -# Disallow: / +User-agent: * +Disallow: /