Technical Web optimization: Technical Search engine marketing refers to Internet site and server optimization that helps the crawler in crawling, indexing and ranking operations to rank your internet site superior. txt file is then parsed and will instruct the robot regarding which web pages will not be being crawled. Like https://www.youtube.com/watch?v=cZppEXx1zPU