While doing SEO, or learning it, many times we have heard a name, robots exclusion protocol (REP) or robots.txt is a text file, webmasters create to instruct robots (typically search engine robots or bots), how to crawl and index pages on their website. Robots.txt are very useful in SEO.

 

Cheat Sheet

Method to Block all web crawlers from all content

User-agent: *

Disallow: /

 

Method to Block a specific web crawler from a specific folder

User-agent: Googlebot

Disallow: /no-google/

 

Method to Block a specific web crawler from a specific web page

User-agent: Googlebot

Disallow: /no-google/blocked-page.html

 

Sitemap Parameter

User-agent: *

Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml

 

Optimal Format

Robots.txt file needs to be placed in the top-level directory of a website on server in order to be useful. Example: http://www.example.com/robots.txt


Pankul Bindal

Pankul Bindal is Digital Marketer, Graphic Designer and Programmer. He has completed his Computer Engineering. His passion is to write blogs, Designing and Digital Marketing. He works for many Companies in his freelancing Career.

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *