Search Engine Optimization – robots.txt and sitemap

Posted on by

Robots.txt is a text file with the rules for search engine robots/crawlers to follow when crawling your website.


robots meta tags are another way to let search engine robots know if are they allowed to crawl the web page.