Summary
Configuration > Settings > SEO Settings has a Page is indexed tick-box on seven page tabs. Unticking it adds the page to your robots.txt (except Homepage), adds a robots meta tag and sends a robots response header. Theme files, download links and single products have their own settings.
Problem
You want search engines to leave out a page of your storefront, such as the search results page or the blog list, or you want your theme files, download links or one product kept out of search results.
Environment
Admin area > Configuration > Settings > SEO Settings, the content editor's File and Image components, and Catalogue > Products > Product Management.
Configuration
Step 1: Open SEO Settings
Sign in to the admin area. In the left-hand menu, click Configuration, then Settings, then SEO Settings. The page opens on the General tab, with a row of page tabs along the top. If you cannot see them all, scroll the tab row sideways.
Step 2: Open the tab of the page you want to hide
Click the tab for the page you want to hide. Seven tabs have a Page is indexed setting: Homepage, Search Results Page, Tag Category Results Search, Registration Confirm, Login Page, Blog Page and News Page. This example uses Search Results Page. The Basket, One page checkout, Order summary and My Account tabs have no such setting: those pages are already kept out of search engines. Page is indexed is normally ticked, which lets search engines list the page; the exception is Login Page, which is unticked, so it is already hidden. Blog Page and News Page cover the blog and news list pages, not individual posts or news items. Be careful with Homepage: unticking it asks search engines to leave your storefront's front page out of their results, which most stores never want.
Step 3: Untick Page is indexed and save
Untick Page is indexed, then click Save at the top of the page. A green message confirms the update and the page returns to the General tab; open the tab again to see your change. This tells search engines not to list the page. Behind the scenes it adds the page's paths to your storefront's robots.txt file as Disallow lines, adds a robots meta tag to the page and sends a robots response header. The Homepage tab adds the meta tag and the header but no Disallow line, so that your whole storefront is not blocked. To let search engines list the page again, tick the box and save.
Step 4: Check the change in robots.txt
In your browser, open your storefront address followed by /robots.txt (for example www.yourstore.com/robots.txt) and reload it with Ctrl+F5 (or your browser's hard-reload shortcut). For the Search Results Page tab you now see these three lines in the list: Disallow: /search$, Disallow: /search? and Disallow: /search/. If your storefront addresses include a language code, each line also appears with the language code in front. Ticking the box again and saving removes them. The file can be kept by browsers and other systems for up to eight hours, so if you do not see the change yet, reload again later. For the Homepage tab, robots.txt does not change: open the page, view its source and look for noindex. Each other tab adds its own paths. Please note: checking robots.txt confirms your setting, not that the page has left a search engine. A search engine can take weeks or even months to drop a page it has already listed, and a page blocked in robots.txt may stay listed. Google Search Console has a removal tool that hides a page sooner, but only for about six months.
Step 5: Check theme files and downloads are hidden
Back in SEO Settings, stay on the General tab. Hide non-image files from search engines is normally ticked; if it is not, tick it and click Save. While it is ticked, your theme's stylesheets and fonts, and download links that do not come from a File or Image component's own setting (for example older templates, or links pasted into text or emails), send a robots response header that tells search engines not to list them. Images are not affected. Files added through a File or Image component follow that component's own setting instead: open the component in the content editor, click the gear icon and set Hide from search engines? (the default is Yes for a File component and No for an Image component). Changing the General setting does not change those components. If a component does not show this option, the General setting applies to it.
Step 6: Hide a single product from search engines
Untick Show in search results on a product to hide it from your storefront's own search and ask search engines not to list it.
- Go to Catalogue > Products > Product Management, search for the product by name, open it and stay on the Product Info tab.
- Untick Show in search results. A new box, Allow direct access when not searchable, appears.
- Check Allow direct access when not searchable, as it may already be ticked. Tick it if customers should still be able to open the product page, for example because the product is in a category or on your home page. Leave it unticked only if customers should not reach the product at all (see below).
- Click Save. If the product is an item of a kit, a warning asks you to confirm the change: click Confirm, otherwise nothing is saved.
- Check it: search your storefront for the product. It should no longer appear.
- To reverse it, tick Show in search results again and click Save.
What unticking Show in search results does
- On your storefront: the product no longer appears in your storefront's search results. It still appears in categories, and on the home page, if you have added it there.
- For search engines: the product page sends a robots response header telling search engines not to list it, and the product is left out of your XML sitemap. Links to it in product boxes are marked nofollow, which tells search engines not to follow them.
- robots.txt: hiding a product adds nothing to robots.txt. The Page is indexed tick-boxes in SEO Settings (step 3) are what add page lines to it.
What Allow direct access when not searchable does (it only appears while Show in search results is unticked)
- Unticked: anyone who opens the product page, from a link, a category or the home page, is sent to your home page, so they cannot open it.
- Ticked: the product page opens as normal. Nothing above changes: the product stays out of search and keeps its robots header.
Note: a search engine that has already listed the product drops it only after it next visits the page, which can take weeks or even months. Searching your own storefront does not show whether a search engine has dropped it.
Search Terms
robots.txt, noindex, page is indexed, search engines, google indexing, hide page from google, stop indexing, seo settings, x-robots-tag, hide non-image files, show in search results, remove page from google, block search engines, noindex page, hide product from google