Posted on

Fill in the blank: The primary function of a robots.txt file is_______________.

  • to prevent pages from appearing in search results
  • to tell search engines about which of your site’s pages you’d like them to crawl
  • to prevent your site from being overloaded with requests by crawlers
  • to control exactly which pages appear in search results