Recall robots.txt will be the part in which you are issuing Directions to the major search engines robots on which directories they need to not check out. The key function with the Robots.txt file will be to guidebook the crawlers and also the bots, about which portion of your site https://rankingseotoolz.com/robots-txt-generator