While doing SEO, or learning it, many times we have heard a name, robots exclusion protocol (REP) or robots.txt is a text file, webmasters create to instruct robots (typically search engine robots or bots), how to crawl and index pages on their website. Robots.txt are very useful in SEO.
Method to Block all web crawlers from all content
Method to Block a specific web crawler from a specific folder
Method to Block a specific web crawler from a specific web page
Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml
Robots.txt file needs to be placed in the top-level directory of a website on server in order to be useful. Example: http://www.example.com/robots.txt