Better Indexing of web sites with Google

Organic traffic, that is, traffic that is generated by people that are searching for your content, is the holy grail of Web advertisement. In order to receive this traffic, web masters have devised even more ingenuous ways of listing their web sites on the top of the listing on Google.

The best way of doing this, however, is helping the web search servers to do their job. Google is trying to make this easier by improving their support of the robots.txt. This is a well known file that determines how the web site is going to submit information to the web search engine.

According to their web site:

The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.

What does robots.txt do?
The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams)
Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.

In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.

Link