The always excellent ContentKing have published another of their in-depth articles.
The post, The Ultimate Robots.txt Guide to Improve Crawl Efficiency, features contributions from:
- Barry Adams from Polemic Digital
- David Iwanow of the Travel Network
- Kevin Indig of Atlassian
- Paddy Moogan from Aira Digital
- Patrick Stox
A robots.txt file is one of those tiny little elements of technical SEO that many sites get wrong. Bookmark that article and save yourself from some avoidable headaches.