No crawl, no follow question!

Question

Hello,Is this possible and will it work or not? Let's say I have a website, but I'd like to have a folder within that site containing a completely different website - however, I would like this "sub-site" not to be crawled by bots / not to be followed.The rule of thumb is you add the robots.txt to the root of the site. By having this robots.txt file in the root of the sub-site, does that count as root or not? - Will this sub-site unintentionally be crawled?Let me know for sure. And also what other suggestions you have for me to achieve this?Thank you.

WolfShade · Accepted Answer

Robots.txt only works with reputable search engines like Google. Others will ignore the robots.txt file and spider your site, regardless. The only way to prevent the sub-site from being spidered is to make it a login/password accessible section.

V/r,

^ _ ^

Nancy OShea · Answer

To be really safe, password protect your folder at the server level. Usually via your server's c-Panel or .htaccess file. Robots cannot crawl a password protected folder.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded