A robots.txt file consists of directives for search engines like yahoo. You should utilize it to avoid search engines from crawling certain areas of your site and to give serps beneficial tips about how they are able to best crawl your web site. The robots.txt file performs a major position https://rankingseotoolz.com/robots-txt-generator