Robots.txt file is a simple text file that is usually placed in the root directory. This robots.txt file can be used to inform the search engine robot to visit / see and index areas that we would want to be consumed by the public and prevent its search engine robots to go into areas that are not for public consumption. Surely not all directory from our website is not for public consumption?

Uses a robots.txt file
Save our Web server bandwidth: robot search engine did not want to visit an area where there is not stored information (eg: cgi-bin, images, area administrators, etc.). If we do not limit, the search engine robots will see areas that are not there and of course information from the web server bandwidth that we use will increase.

Besides robots from search engines generally do not want to linger and some of them have a time limit to visit a website. Maybe if we did not use robots.txt to content published yet seen by the robot, but robots have time to visit a web site is up, this may be also used as a reason why search engines prefer a simple website.

Can be used as an initial protection: I’ve described above. With the robots.txt can be used as initial protection to the privacy page for us not in the consumption by the public.