When learning about SEO, the robots.txt file is likely to come up. This is a powerful file that ‘speaks’ directly to Google and Bing (& Other) search engine robots.
Is my Robots.txt Correct?
It is very important to understand what this file can do. The robots.txt file seems to generate a lot of opinions but you must truly be sure about your editing as this can start and stop Google from indexing your site!
- the order of the instructions matter
- Google wants to see everything (less is more)
- only disallow if there is a very specific reason for it (ie: affiliate links)
What to do if your robots.txt file is wrong
If you’re quite sure that the robots file is wrong, check Yoast at this article for clear instructions on changing it. But I can’t not say it again: be careful!