Wednesday, October 19, 2005

Search Engines

Working out a way to stop crawler to fetch information from one of my sites, it's HERE for Google.

Placed the code META NAME="ROBOTS" CONTENT="NOINDEX, NOARCHIVE, NOFOLLOW" between head tags in the class domain name, so that it won't be indexed.

I have also placed a robots.txt file on the root of my server to stop all crawlers. Robotstxt.org tells you how to do that.

PS: Google offers the service of immediately remove your pages from their index HERE but this link isn't working for some reasons. Yahoo doesn't do this.

No comments:

Post a Comment