There are a thousand reasons why you might not want search engines indexing particular content on a published project. Maybe you need to publish sensitive information but don't want Google — or any other search engine — listing that page in search results.
In this lesson
We're going to show you one of the most common ways to block Google from indexing your content — using this snippet of code: <meta name="robots" content="noindex">
Before you begin
- You can remove the webflow.io subdomain from the index by disabling indexing in your Project settings.
- If you’re comfortable with writing robots.txt files, you can also disable indexing of pages, folders of pages, or the entire site by populating the project’s robots.txt file.
Remove a page from the index
To remove a specific page, add the code snippet in the <head> code of that page</head>.
- Go to Page panel → Page settings → Custom code
- Paste the code: <meta name="robots" content="noindex">
- Save and publish
Remove a site from the index
To remove the entire project from the index, add the same code in the <head> code of the project</head>.
- Go to Project settings → Custom code → Head code
- Paste the code: <meta name="robots" content="noindex">
- Save the changes
- Publish the site
Good to know
- You won't need to add the noindex code for pages that are saved as draft since they aren't published at all. So, if you have any pages that are still under construction, or that you do not need to publish (like your project’s style guide for example), it's best to save them as draft.
Troubleshooting
Search engines must be able to crawl your page in order to see the noindex meta tags. If they cannot crawl your page, they won’t be able to read that code, meaning your page will still be indexed and might still appear in search results.
If your page is still appearing in search results after you’ve added the noindex code, check the following:
- Go to your page settings and check that the code snippet is saved in the head section of your custom code
- Publish the site again
- Request Google to recrawl your page using the Fetch as Google tool
- Check your robots.txt file if it's blocking your page or site. You can test your robots.txt using the robots.txt Tester tool.
- If the page is blocked from webcrawlers, unblock it by removing it from your robots.txt file
- Publish your site again
- Again, request Google to recrawl your page
