Blocking Search Engine Indexing

Blocking Search Engine Indexing

This article explains how to prevent search engines, such as Google, from crawling and indexing your website. This is especially useful for demo, staging, or non-public sites that you do not want to appear in search results.

There are two primary methods to block search engine indexing: using a robots.txt file to disallow crawling and adding a noindex meta tag to your site’s HTML. Both methods can be easily implemented to ensure your site remains hidden from search engine results.

In addition to these settings, you can also password-protect your website for an extra layer of privacy. Once you have applied your chosen method, you may need to request that search engines re-crawl your site to update their indexing.

Use cases

• Demo or staging sites that are not intended for public discovery.
• Websites that contain sensitive or under-development content.

Key settings in Infigo

• PageHeadContainer: Found in Infigo Settings; add the noindex meta tag snippet to apply to every page of your storefront.
• robots.txt file: Place this file in the root directory of your website to disallow all search engine crawlers.

Step-by-step guide

1. Decide whether you want to use a robots.txt file, a meta tag, or a combination of both to block indexing.
2. To use a robots.txt file, create or update the file at your website's root (for example, https://example.com/robots.txt) with the following lines:
   
   User-agent: *
   Disallow: /
3. To use a meta tag, add the following snippet to the PageHeadContainer field in your Infigo Settings:
   
   <meta name='robots' content='noindex'>
4. Save your changes and, if necessary, request that Google recrawls your site so that the new settings take effect.

For additional guidance, head over to the Infigo Academy or contact our Customer Support team.