Resources
How do I disallow BrandProtect
robots
from spidering my web site?
How do I disallow BrandProtect robots from spidering my web site?
To disallow BrandProtect robots from crawling your site. Please add the robots.txt to the root of your web site hierarchy
(e.g. www.example.com/robots.txt).
The robots.txt file should contain the following lines:
User-agent: robot-name
replace the 'robot-name' with the exact robot signature you found in your web log
Disallow: /directory/
the top directory from which you do not want the robot to crawl
For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com would not apply to a.example.com. You should add each distinctive robot signature to the file.
BrandProtect helps its clients search for their digital assets on the internet. Our robots are programmed to avoid overloading any web site with too frequent requests and respect instructions found in robots.txt. There are times when our clients request that a particular site be scanned and this may increase traffic to your site on a one-time basis.
Again we understand your concern and thank you for reaching out to remediate the situation. If after adding the robot signatures to the robots.txt file and yet our robots still do not behave accordingly, please contact us.
You will find additional information at:
http://en.wikipedia.org/wiki/Robots.txt