Use robots.txt to Increase Search Engine Rankings
Filed in: SEO Beginner — March 8th, 2007
According to a Link Building Blog post by Neil Patel, he uses a robots.txt file to remove junk pages and duplicate content from the search engine index databases. As a result, his website traffic went up 11.3%!
Here are the things he did with his website:
- Removed comment feeds from search results so that no duplicate comment text indexed.
- Removed trackback URLs from being indexed because it was causing blank pages to be indexed.
- Denied search bots access to his blog installation folder (MovableType).
Here is Neil Patel’s robots.txt file content:
User-agent: *
Disallow: /mt
Disallow: /*.xml$
Disallow: /*.cgi$
A simple file with four like of codes! However, it is for MovableType blog only. For WordPress users, askApache prepared the robots.txt files for WordPress and phpBB users. Great!
My two cents
robots.txt file is a plain text file that control the search bots access to your website. I wonder why SEO gurus seldom talk about it. I will try this tip and see the result. 
But, please be careful when you apply robots.txt. A wrong line of code can block ALL search bots from indexing your website!
Popularity: 2% [?]
Comments Feed
TrackBack URL
March 8th, 2007 at 4:48 pm
Interesting! I will try this also. I will be glad if you could post your results after some time – ok?
March 11th, 2007 at 8:33 pm
[...] Use robots.txt to Increase Search Engine Rankings By LewisCF , Four line trick to kick start seo optimization for your blog. [...]
April 16th, 2007 at 9:09 am
how is the result?
April 16th, 2007 at 1:13 pm
my blog feed url has lower rank now. Previously in 1st page, now in 2nd page.
April 19th, 2007 at 7:20 am
would you mind to post the robots.txt that you use?
does it bring more traffic to your website?
May 25th, 2007 at 11:01 pm
There is a new and updated robots.txt example at http://www.askapache.com/seo/wordpress-robotstxt-optimized-for-seo.html