txt file, Alternatively, tells search engines like google and yahoo what material never to index (as an example, particular policy webpages you don’t want to appear in look for.) produce each to speed up the crawling https://medium.com/@linkdaddy