The robots.txt file is a small file located in the root folder of your Joomla site. The file contains instructions to the search engines on what to index and what to leave out.
Some people confuse the robots.txt file with the .htaccess file. The difference is significant. The first one only gives instructions to search engines - and most search engines respect it. The robots.txt file is not used for anything related to security. Nevertheless, it can have an impact on Search Engine Optimization.
The .htaccess file, on the other hand, is used to reconfigure the settings of your Apache server, redirect URLs and other server related tasks. More about that in a later post.
The robots.txt file and SEO
As mentioned, the robots.txt file is in your site root folder. It contains info on which folders should be indexed and not. I can also include information about your XML sitemap.
There are just two tips I would recommend regarding SEO and the robots.txt file:
1. Remove exclusion of images
For reasons I don't understand, the default robots.txt file in Joomla is set up to exclude your images folder. That means your images will not be indexed by Google and included in their Image Search. And that's something you would want, as it adds another level to your sites search engine visibility.
To change this, open your robots.txt file and remove the line that says:
Disallow: /images/
By removing this line, Google will start indexing your images on the next crawl of your site.
2. Add a reference to your sitemap.xml file
I've talked about the Sitemap XML file previously, in my post on How to get your Joomla site indexed in Google. If you have a sitemap.xml file (and you should have!), it will be good to include the following line in your robots.txt file:
sitemap:http://www.domain.com/sitemap.xml
Naturally, this line needs to be adjusted to fit your domain and sitemap file. In my case, I use the Xmap component to create the Sitemap XML file automatically.
So, the line looks like this for Joomlablogger.net:
sitemap:http://www.joomlablogger.net/component/option,com_xmap/lang,en/sitemap,1/view,xml/
Other than that, the robots.txt file can live happily at peace in your site's root folder.
UPDATE: Please note that there are some changes to the robots.txt file, starting with Joomla 3.3.