What is Robots.txt?

While doing SEO, or learning it, many times we have heard a name, robots exclusion protocol (REP) or robots.txt is a text file, webmasters create to instruct robots (typically search engine robots or bots), how to crawl and index pages on their website. Robots.txt are very useful in SEO.

 

Cheat Sheet

Method to Block all web crawlers from all content

User-agent: *

Disallow: /

 

Method to Block a specific web crawler from a specific folder

User-agent: Googlebot

Disallow: /no-google/

 

Method to Block a specific web crawler from a specific web page

User-agent: Googlebot

Disallow: /no-google/blocked-page.html

 

Sitemap Parameter

User-agent: *

Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml

 

Optimal Format

Robots.txt file needs to be placed in the top-level directory of a website on server in order to be useful. Example: http://www.example.com/robots.txt

Leave a Comment

want more details?

Fill in your details and we'll be in touch