Knowledge Base


  • Video block
  • Video Tutorials

  • Watch our easy-to-follow video tutorials for quick tips on designing and managing your store.
  • View Video Tutorials

Configuration: General - Design - Search Engine Robots

System > Configuration > General > Design > Search Engine Robots


This section lets you customize a robots.txt file to control which pages of your site are indexed by search engines. The robots.txt file is a text file that is checked whenever search engine robots visit your website. Search engine robots, or search engine “bots,” are machines that index web content, including your store. The robots.txt file is currently the standard for determining which information on a website gets indexed. When a search engine robot visits your website, it first looks for a robots.txt file. If one is found, the search engine robot follows the instructions in the file.

Field Descriptions

Field Description
Default Robots Select a default action for search engine robots from the list:
  • INDEX, FOLLOW: Pages are indexed, and search engine bots are allowed to follow links from applicable pages.
  • NOINDEX, FOLLOW: Pages are not indexed, but search engine bots are allowed to follow links from applicable pages.
  • INDEX, NOFOLLOW: Pages are indexed, but search engine bots do not follow links.
  • NOINDEX, NOFOLLOW: Pages are not indexed, and search engine bots do not follow links.

Note: Applicable pages are those that are not excluded using the “Disallow” feature of the robots.txt file.

Edit custom instruction of robots.txt File Select instructions to add to your customized file from the list. The first instruction, “User-agent: *” tells the robot that the rules listed below it apply to all robots. The “Disallow: /” instruction followed by a page designation tells the robot not to visit the page specified in the instruction. These instructions are included before the head closing tag in the page HTML.
Reset to Default Set to Yes or No. Selecting Yes deletes your custom instructions and resets the robots.txt file to default settings.

See Also:

Setting Up a Robots.txt File