Lately I happened to check out the usage of robot.txt to search for any help on SEO optimized. Robots.txt also known as The Robots Exclusion Protocol is a set of instruction created for web robots or spiders. The search engine crawler will look for robots.txt where pages to indexed and ignored are defined.
If you have a blog and trying to locate your robots.txt file, most probably you won’t find it. That’s because in order to allow full access for all web robots or spiders, not having a robots.txt is a way to do it. That is why our blogs still get index even without a robots.txt. If that is the case, do we actually need to add robots.txt file?
It is said that robots.txt is use to avoid issues like duplicate content getting index. The intention of having robots.txt is to help improved page index and getting higher page rank. If you don’t really understand how robots.txt works, it’s better to not to touch it as it might do more damage than help. I did some study and research on robots.txt for some time and still couldn’t quite get it myself. If you still insist on trying it out, get the PC Robots.txt WordPress plugin. What it actually does is creates a virtual robots.txt that is east to edit and manage. Once activate, you’ll be given a default robots.txt that disallowed a bunch of spam-bots and some standard WordPress folders and files. Feel free to mess around with the robots.txt. You can always return to the default setting by reactivating the plugin. You can do all this right from your blog admin.
You can easily get information on what you can put into the robots.txt file but there isn’t any post telling what configuration is best for SEO. I guess it’s up to us to try it out. Here’s a post form SEOmoz that I think is good to check out as it explains how robots.txt affects link juice.
P/S: After trying out robots.txt myself. I think it’s better not to do anything about it. If you guys found it differently, please share.