Individuals use DA to check out which Internet websites are trustworthy within their subject and to determine wherever to acquire very good links. txt file is then parsed and will instruct the robot as to which internet pages will not be for being crawled. Like a search engine crawler could https://www.youtube.com/watch?v=imiGL2YS2uo