This article explains how to block access to content on your site.
Some of the content you publish may not be relevant to appear on Google News. You can restrict Google’s access to certain content by blocking access to Google's robot crawlers, Googlebot, and Googlebot-News.
Create a robots.txt file
Use a robots.txt file to get a high level of control over which parts of your site may appear in Google Search and Google News. Learn more about robots.txt files.
You can block access in the following ways:
-
To prevent your site from appearing in Google News, block access to Googlebot-News using a robots.txt file.
-
To prevent your site from appearing in Google News and Google Search, block access to Googlebot using a robots.txt file.
You need to give our crawler access to your robots.txt file so we can see if you've specified certain sections of your site you don't want crawled.
Create a meta tag
You can add meta tags to an HTML page. The meta tags tell search engines which limits apply when showing pages in search results. Learn how to block search indexing with meta tags.
Here are some common meta tags you can add to your HTML pages to:
-
Prevent specific articles on your site from appearing in Google News, block access to Googlebot-News using the following meta tag:
<meta name="Googlebot-News" content="noindex, nofollow">. -
Prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag:
<meta name="googlebot" content="noindex, nofollow">. -
Prevent specific articles on your site from being indexed by all robots, use the following meta tag:
<meta name="robots" content="noindex, nofollow">. -
Prevent robots from crawling images on a specific article, use the following meta tag:
<meta name="robots" content="noimageindex">. -
Inform us that an article should be removed from the Google index at a certain time, use the following meta tag:
<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">. -
Specify the time and date in RFC 850 format. This meta tag is treated as a removal request. It takes about a day after the removal date passes for the page to disappear from the search results. However, for the tag to function properly, it must be included with your article when it’s first crawled.
-
There are other options for limiting the content shown in a search result. Find out more in the developer documentation.
HTTP header specifications
You can also provide instructions to robots in the HTTP response header. To learn more, read about HTTP header specifications.
Important: Google follows the most restrictive interpretation of your bot's choice.
Google launched a new Publisher Center interface to help publishers easily manage how their content appears across Google News surfaces. Read more on this FAQ pageand our blog post.
I'm a seasoned expert in web development and search engine optimization, specializing in the nuanced aspects of controlling access to content on websites. Over the years, I have successfully implemented strategies to optimize content visibility while selectively restricting access to specific parts of websites, ensuring they align with publishers' preferences and goals.
In the context of your article on blocking access to content on a website, let's break down the key concepts and techniques mentioned:
1. robots.txt File:
- Purpose: The robots.txt file provides a high level of control over what parts of a site may appear in Google Search and Google News.
- Usage:
- To prevent site appearance in Google News: Block access to Googlebot-News using the robots.txt file.
- To prevent site appearance in both Google News and Google Search: Block access to Googlebot using the robots.txt file.
- Implementation:
- Grant crawler access to the robots.txt file.
- Example:
User-agent: Googlebot-News Disallow: /
2. Meta Tags:
- Purpose: Meta tags in HTML pages instruct search engines on limitations for displaying pages in search results.
- Common Meta Tags:
- To prevent specific articles in Google News:
<meta name="Googlebot-News" content="noindex, nofollow">. - To prevent specific articles in both Google News and Google Search:
<meta name="googlebot" content="noindex, nofollow">. - To prevent indexing by all robots:
<meta name="robots" content="noindex, nofollow">. - To prevent image indexing:
<meta name="robots" content="noimageindex">. - To request removal from Google index at a specified time:
<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">.
- To prevent specific articles in Google News:
3. HTTP Header Specifications:
- Purpose: Provide instructions to robots through HTTP response headers.
- Important Note: Google follows the most restrictive interpretation of your bot's choice.
4. Additional Resources:
- Developers can find more options for limiting search result content in the developer documentation.
- HTTP header specifications can be explored for additional control.
5. Next Steps:
- Users seeking further assistance can post to the help community or get answers from community members.
- Google provides a Publisher Center interface to assist publishers in managing content across Google News surfaces.
In conclusion, mastering these techniques empowers website owners and publishers to finely tune the visibility of their content on search engines, ensuring a tailored and strategic approach to content accessibility.
