Dignity nobility high-quality point out highness standing class caste status superiority precedence fashion difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot regarding which webpages are not to get crawled. Being a online search engine crawler might hold https://www.youtube.com/watch?v=jfBrROL69Lk