How to Optimize WordPress Robots.txt File for Better SEO
I do recommend, however, that you disallow the readme.html file in your robots.txt file. This readme file can be used by someone trying to figure out which version of WordPress you’re using. If this is a person, they can easily access the file by just browsing to it. Plus, putting a disallow tag in can block malicious attacks.
How to Optimize Your WordPress Site Robots.txt for SEO?
You need Robots.txt to tell search engines which pages or folders shouldn’t be crawled. If you don’t have it, the search engine will still crawl the and index your website. It doesn’t affect a lot to your site. But, when your website keeps on growing and you have lots of content in your site, it becomes unwantedly compulsory for your site too.