What are we likely to see when the WordPress 5.3 release becomes a reality?

• Beta 1: 23 September 2019
• Release Candidate: 15 October 2019
• General Release: 12 November 2019

The great thing about WordPress is that it’s always evolving, so, typically, the WordPress community are already weighing in on what the WordPress 5.3 release has in store.

With new and improved features in the Gutenberg page editor, here are three things you are very likely to see when version 5.3 becomes a reality.

Sites with the “discourage search engines from indexing this site” option enabled will display an updated robots meta tag to prevent the site from being listed in search engines: <meta name='robots' content='noindex,nofollow' />.

Blocking Google From Indexing

The word “indexing” meant crawling of the site by GoogleBot. By using the Robots.txt blocking feature you could stop Google from downloading the specified web page and, it was assumed, Google would be unable to show your pages in the Search Results.

But that robots.txt directive only stopped Google from crawling the page. Google was still free to add it to its index if it was able to discover the URL.

So to block a site from appearing in the index, a publisher would block Google from “indexing” the pages. Which wasn’t consistently effective.

WordPress 5.3 Will Truly Prevent Indexing

WordPress adapted the Robots.txt approach. But that’s changing in version 5.3.

When a publisher currently selects “discourage search engines from indexing this site” what that does is add an entry to the site’s robots.txt that prohibits Google from crawling a site.

Starting with WordPress 5.3, WordPress will adopt the more reliable Robots Meta Tag approach for preventing the indexing of a website.

This change will affect the “discourage search engines from indexing this site” setting.

This change is an improvement. WordPress publishers can be more secure in knowing that the blocked web pages will not be shown in Google’s search results.

Why Did WordPress Use Robots.txt?

WordPress relied on Robots.txt for blocking the indexing of a website because that’s how everybody kept pages from showing in Google’s search results. That was the standard way of doing it.

Yet even though everybody did it that way, as has been explained, it was an unreliable approach.

The word “indexing” has two meanings:

1. Indexing means crawling, as when Googlebot visits and downloads web pages.
2. Indexing can also mean adding a web page to Google’s database of web pages (which is called The Index).

Blocking Google from “indexing” a web page will keep it from seeing the web page but Google could still index the web page and add it to Google’s index. Make sense?

Robots.txt Versus Robots Meta Tag

So it’s good to see WordPress embrace the Robots Meta Tag as the solution to blocking web pages from showing in the search engines.

Read the WordPress Planning Roundup:

Changes to Prevent Search Engines Indexing Sites

Read the WordPress announcement:

Changes to Prevent Search Engines Indexing Sites