Monday, February 6, 2006

Stop Crawlers

Have read this again as to how I can stop crawler to look into a particular folder of my site. Firstly you could only have one robots.txt and that should be at the root of your html folder.

User-agent: *
Disallow: /emba/

By this I will stop crawler to look into just one subfolder /emba/. My post detailed earlier setup, i.e. used just "disallow: /" which means entire site will not be crawled/indexed.

I have also removed the index.html file at the root which redirects visit to My EMBA if someone hits just

What I will do is to upgrade my current plan to multi-site then I could make the class blog to run under its domain name without redirection. Currently a domain name pointing is setup so visitors will be here if they use

No comments:

Post a Comment