Should i disallow cgi bin

The preferred way of preventing parts of your site from being indexed is to use the Control Center page exclusion mechanism. This is covered in How to Exclude Pages from Search. You should read that "how to" first. The only reason you might need to use a robots. Contents Overview Examples Reference A "robots. Each disallow will prevent any address that starts with the disallowed string from being accessed.

Using a robots. It can also be used to refer to all robots. This will include all folders contained in it. Specifying certain bots is also allowed and in most cases very useful to users that utilize doorway pages or other ways of search engine optimization. Listing individual bots will allow a site owner to tell specific spiders what to index and what not to index. White spaces and comment lines can be used but are not supported by most robots. When using a comment, it is always best to add it to a new line.

Notice on the first one the comment line is on the same line indicated by a then the comment. While this is ok and will be accepted in most cases, a lot of robots may not utilize this. So be sure to use Example 2 when using comments. White spaces refer to using a blank space in front of a line to comment it out.

It is allowed but not always recommended. Note also that globbing and regular expression are not supported in either the User-agent or Disallowlines. What you want to exclude depends on your server. Everything not explicitly disallowed is considered fairgame to retrieve.

Here follow some examples:. This is currently a bit awkward, as there is no "Allow" field. Theeasy way is to put all files to be disallowed into a separatedirectory, say "stuff", and leave the one file in the level abovethis directory:.

English US. US English US.