I want Google to ignore my "staging" site

[lateral]

Hi guys

 

I have a "staging" "test" domain set up for my shop.

 

I do not want Google or any other crawler to crawl it or even know about it.

 

is there a simple way to guarantee that they can't crawl it???

[parsifal]

Hello lateral.

 

 

1. You could use a robots.txt file: http://www.robotstxt.org/faq/prevent.html. But there is no guarantee here. Webcrawl bots can ignore the robots.txt file, if their creator chooses to do so.

 

2. You may be able to configure your web server so it denies HTTP requests with user agent headers different than those of well-known browsers. This is also not a sure way, since some bots could easily masquerade themselves as a browser.

 

3. You could implement a simple username-password authentication at the web server level. Example: http://www.siteground.com/tutorials/cpanel/pass_protected_directories.htm. This would stop virtually all bots.

[lateral]

Thanks for the ideas....

[lateral]

I set up a password protected HTACCESS file.

 

Works great!