Next, you must make your mind up how you desire the crawler to manage various Net technologies. There is an ongoing discussion concerning the intelligence of search engine crawlers. It isn't really completely obvious When they are full-blown headless browsers or just glorified curl scripts (or a little something https://www.youtube.com/watch?v=EVeTP0aV8gY