{Update: Google’s John Mueller replied to my question on Twitter about this situation. You can read John’s feedback here.}

I’ve been helping a client with a large-scale CMS migration recently and came across a very interesting, and potentially scary, situation. It’s a large and complex site with many moving parts, so there is a lot to process (both figuratively and literally).

The site has over one million pages indexed, and many more that get crawled based on the proper use of noindex in certain areas of the site. That was a big change for the site when I first started helping them a few years ago (after they got pummeled by a major algorithm update). Therefore, I’m always making sure those areas of the site are handled properly (as changes and updates go live on the site).

And now that a CMS migration was under way, I made sure to check on those areas of the site to make sure all was ok. It ends up they were ok… but they sort of weren’t either. Confused? Then read on.

The Magic Show Begins – Typical checks after a CMS migration
As I started crawling and checking the site after the migration went live, I noticed something very strange. The number of noindexed pages during my crawls was sometimes lower than what I expected. And while manually checking those pages, I noticed that my trusty AYIMA plugin sometimes went red (indicating the page was not indexable), but sometimes it didn’t. I found that odd and began to dig in further. Note, other plugins like the DeepCrawl indexability plugin also showed those urls were indexable. Again, strange…

Static HTML Correct, Rendered HTML Not So Much…
When checking the areas of the site I mentioned earlier, and digging into the source code (the static html), I noticed that the meta robots tag was in the proper place and contained the correct directives. For example, it was placed in the head of the document and contained “noindex, follow”.

But we know that Google renders the page and that rendered html can be different than the static html. For example, JavaScript-based content will be rendered and can be indexed by Google. So, I started to check the rendered html via a number of tools like Google’s mobile-friendly test, the rich results testing tool, Chrome Dev Tools (using inspect element), and then various crawling tools like DeepCrawl, Screaming Frog, and Sitebulb. They have the ability to render the page, revealing what the final rendered html would reveal.

Note, the rich results testing tool will render the page via Googlebot desktop, while the mobile-friendly testing tool will render the page as Googlebot for Smartphones. Chrome Dev Tools will show you the rendered html via “inspect element” and then crawling tools like Screaming Frog, Sitebulb, and DeepCrawl have options for crawling with JavaScript rendering on. Therefore, it’s good to test across all of the tools to see what the results are across both mobile and desktop.

Render JavaScript option in Screaming Frog (first image) and then DeepCrawl (second image):

Wait, what just happened?
When I checked the rendered html across those tools, I witnessed the magically moving meta robots tag. The robots tag, and some other head elements, were now in the body of the document and not the head. I had to double and triple check this to make sure I was seeing it properly.

The meta robots tag went from line 105 in the source code to line 1,181 in the rendered version of the code. And again, it was NOT in the head of the document, where it needs to reside. It was now in the body! Here is a screenshot from Screaming Frog showing both the source html and the rendered html. Notice the line numbers next to each meta robots tag:

But Is It Safe?
When spot-checking urls containing the magically moving meta robots tag, they indeed were noindexed. Therefore, I’m assuming Google is using the meta robots tag from the static html and not in the rendered html (more about this in the next section). But will that always be the case? Again, on a site with millions of noindexed pages, and a history of getting hit by major algorithm updates, is it safe to believe that the magically moving meta robots tag won’t eventually cause problems? All good questions.

Google’s 2-phase approach for indexing JavaScript-based content
I passed all of this information along to my client and that was sent to the dev team. I’m still waiting to hear back about why this is happening, but needless to say, it could be very dangerous on a large-scale site with many moving parts. In addition, this is happening in sections of the site that absolutely should be noindexed and not released into the SEO wild.

Also, Google has explained that it uses a two-phase approach for indexing JavaScript-based content. The first phase will crawl and index the static html. And the second phase, which can take a few days or longer, will process and index the rendered html. And remember, the rendered html is where the meta robots tag ends up in the body of the document (for this case).

Here is John Mueller explaining that inserting canonical tags or meta robots tags via JavaScript can cause issues based on the initial phase of indexing versus the second phase (where it renders and indexes the JavaScript version):

So, theoretically for my client, Google would first see and process the proper meta robots tag in the static html, but then it would process and index the rendered html, which has the meta robots tag in the body of the document. If that’s the case, then Google may not decide to noindex the page in the long-term. I have to dig in further across those areas of the site to see how many pages are being noindexed versus indexed over the long-term (even though I’m hoping we just fix this problem sooner than later so we don’t need to worry about this!)

Again, with millions of pages currently being noindexed, I would NOT want to see a chunk of those pages suddenly get indexed. Remember, this site has felt the wrath of major Google algorithm updates in the past. It’s doing very well now, but the battle scars remain.

The CMS or the implementation?
Since my client just moved to a new CMS (a popular enterprise CMS), I naturally starting wondering if the problem was CMS-based or if it was due to the implementation of the CMS. Companies typically tailor the CMS implementation with various customized elements and code, and that can always cause unforeseen issues.

So, I decided to check several others sites running the CMS and they looked fine. The meta robots tag was on the same line in the html in the static and rendered versions (and it was located in the head of the document). Therefore, this looks to be an implementation problem. That’s often the case with CMS migrations. It’s not necessarily the CMS, it’s more about how the implementation goes. That’s where bugs can creep in and potentially cause big problems SEO-wise.

Update: Feedback From Google’s John Mueller
John Mueller replied to my question about this situation today on Twitter. John explained that if the meta robots tag is in the right place in the static html, then that should be fine. He explained that it’s different when you need to do something specific after indexing and rendering (like adding hreflang via JavaScript). That’s great to know. I am still going to make sure the problem is fixed, since I’m not a fan of having magically moving meta robots tags, but it’s great to know this shouldn’t be a big problem. That makes sense, since I am seeing those pages accurately noindexed. Here’s John’s tweet:

Quick Tips For Stopping Magically Moving Meta Robots Tags
Based on this case, I have some closing tips for SEOs and site owners that are going through a CMS migration (or for anyone that wants to make sure the meta robots tag, rel canonical, or other important SEO code is set up correctly).

• Test your site via multiple tools that enable you to view the rendered html. That includes the mobile-friendly test (mobile), the rich results testing tool (desktop), Chrome Dev Tools (using inspect element), etc.
• You can also test your site via multiple crawling tools that render JavaScript, including Screaming Frog, DeepCrawl, and Sitebulb. Then check the static html versus the rendered html. And make sure to check important reports to see what has changed compared to previous static crawls. For example, you might see noindexed pages drop off a cliff when you compare a static crawl to a JavaScript-rendered crawl. If so, why?
• If it’s a third-party CMS (and not custom built by your company), then test other sites running the CMS. You might see consistent behavior across those others sites, which would signal a problem with the CMS itself. But if they look fine, then it’s probably your implementation of the CMS. That’s important to know.

Summary – For SEO, don’t rely on magically moving code
This was an interesting case for sure. Any time you see important directives magically moving around the html code, you definitely want to dig in and find out what’s going on. And then it’s important to quickly address the situation with the dev team. That’s especially the case for large-scale sites with many moving parts. I’ll try and post an update after my client surfaces the problem and applies a fix. Needless to say, I’m hoping the magic show ends soon. :)

GG

Based partly on the August 1 Google algorithm update, there’s been an extreme focus from site owners and SEOs on building and demonstrating E-A-T (expertise, authoritativeness, and trust). Now, the concept of E-A-T has been documented in Google’s quality rater guidelines (QRG) for years, but the recent update did seem to have an element that turned up the volume from an E-A-T perspective (and especially for sites focused on health and medical.) That’s why Barry Schwartz decided to name it the Medic Update.

Keep in mind that many different types of sites were impacted by the August 1 update and Google even confirmed they didn’t target sites that could be categorized as Your Money or Your Life (YMYL). Also, it’s important to know that you can’t build E-A-T quickly. I’ve mentioned that in my previous posts about major algorithm updates. You can, however, build strong E-A-T over time if you are doing the right things (building great content written by experts in your niche, using Social to reach a targeted audience, naturally building links from well-known sources, etc.)

Regarding the Quality Rater Guidelines (QRG), Google’s quality raters are asked to evaluate its algorithms, and E-A-T comes up many times in the 160-page guide. Note, raters cannot directly impact the rankings of any specific site, but their feedback is sent to the engineers, who then refine their algorithms. Therefore, they sure can impact rankings indirectly.

Based on what I explained above, site owners tend to have a few important questions:

• Is my site and content trusted?
• Are my writers perceived as experts in a niche?
• Which specific factors on my site would cause users (and Google) to not trust the site?
• Is there any way to see ratings for my site?

When helping companies that have been significantly impacted by a major Google algorithm update, it’s important to review the site overall, identify all problems that could be causing issues, and form a remediation plan for fixing those problems as quickly as possible. And helping companies determine perceived E-A-T is important. For example, do you have expert authors, are you deceiving users, does your content exhibit a high level of authoritativeness, etc.

The topic is extremely nuanced, since each niche category is different. For example, a medical site is different from a financial site, which is different from a dating website. And those sites are much different than coupon sites or lyrics websites. Again, there are many different types of sites on the web. And to address the last question from above about actually seeing ratings, there’s no way to see what raters thought of your site while evaluating Google’s algorithms.

That’s unless you’re a news publisher… Did I get your attention? :)

Well, you can’t see what Google’s quality raters think, but you can see view what Newsguard’s team of analysts think. And that can be a good proxy for understanding issues your site has regarding trust, credibility, and transparency.

Newsguard – Publicly evaluating trust and transparency of news publishers
Based on what I explained above, it’s great when you can get objective third parties to review a site (almost like your own version of Google’s quality raters). You can do that on your own, but there’s a good amount of work to do in order to run effective user studies. That’s why it’s great when you can find public sources of information from other services that can yield strong feedback about your site. That’s exactly what Newsguard is doing (for news publishers).

On its site, Newsguard explains that it aims to fight fake news, misinformation, and disinformation. I’ll explain more about Newsguard in the next section, but think of it as a way for the average person to quickly identify organizations that are adhering to important journalistic standards, and not participating in misleading the public.

Newsguard has a Chrome plugin that you can install that provides a quick way to see ratings for news publishes across Search and Social Media. You can also see ratings when visiting the sites in question. As soon as I read an article about Newsguard, I had to check it out. And after using it, I found it fascinating to receive objective feedback about the trustworthiness of various news organizations. It had me wishing there was a Newsguard for every category of website.

How It Works – Criteria and Nutrition Label
Newsguard has analysts that are trained journalists objectively rate news publishers based on nine criteria broken down into two major categories, credibility and transparency. Each criterion is weighted and worth a certain number of points. If a site receives 60 points or higher, it gets a green label (which is displayed in the browser, in the Search results, and across Social Media sites). If it falls below 60, it gets a red label. There are also other labels for user-generated content, satire and humor sites, etc.

For example, here’s what the plugin looks like in the browser window:

And here’s what it looks like in the Search results:

And here’s what it looks like across Social Media sites:

And you don’t just receive basic feedback (like the score). You can see a full “nutrition label” to learn more about why the site received that score, what the analysts listed as reasons for the scoring, etc. Newsguard explains they are completely transparent about why a site received a certain score. They also reach out to the sites being reviewed for more information and will publish that too. And if a score needs to be adjusted based on new information, Newsguard will document that information as well. You can read more about their corrections policy on the site.

For example, here’s an example of a nutrition label for CNN:

And you can read the full rating with information from analysts:

9 Factors Evaluated By Analysts
You can read more about the criteria on the Newsguard website, but they look at a number of factors that are important for any news publisher (and it’s important to note that there’s overlap with certain parts of Google’s quality rater guidelines).

For example, from a credibility standpoint, does the site:

• Publish false content.
• Gather and present information responsibly.
• Correct or clarify errors.
• Handle the difference between news and opinion responsibly.
• Avoid deceptive headlines.

And from a transparency standpoint, does the site:

• Disclose ownership and financing.
• Clearly label advertising.
• Reveal who is in charge, including conflicts of interest.
• Provide information about content creators.

Again, each factor is scored and weighted, with 60 points being the difference between a green label and red label. You can see the weighting below:

And for those of your reading this post involved in SEO, your QRG antennae might have gone up several times when reading those bullets points. For example, they mention deceptive advertising, bios for content creators, deceptive headlines, and more. Again, wouldn’t it be great to have this objective feedback as a starting point for every site on the web?

Examples of Site Ratings:
Below, I’ll provide a few examples so you can quickly see two sites on opposite ends of the spectrum. One receives very strong ratings, while the other sits below the 60-point threshold.

The New York Times (view the page on Newsguard.com):

A news publisher with 6 major categories of content, including YMYL topics, that received a red badge (less than 60 points):

Not listed? Submit your site.
As you can guess, the major limitation here is that this is a very manual process for Newsguard. That’s why I came across a number of news publishers that were in the process of being reviewed, or not even in the system yet. I don’t know how long it takes analysts to review each site, and then how long it takes for editors to review those ratings, and then for the co-CEOs to make a final review, but the process doesn’t seem quick. I’m not saying it should be quick, but I can see them getting swamped with requests very quickly. You can read more about the review process on the site.

So if you don’t see a rating for your news site, then feel free to submit it to Newsguard. I submitted a few dozen during my testing. It should be interesting to see how long it takes for those submissions to be reviewed. I’ll share more on Twitter when I see some of those submissions finally receive scores.

Here’s an example of a site in the process of being reviewed:

A Note About Google’s Major Core Algorithm Updates – How do sites with failing grades fare during major updates?
While testing Newsguard across the web to check news publishers, I kept wondering how good of a proxy it was for Google’s quality rater guidelines (and how that would manifest itself during major algorithms updates).

For example, would failing sites also see negative movement during broad core updates? Let’s face it, publishing false information, providing deceptive advertising, using deceptive headlines, and more certainly are things that could cause algorithmic problems. But remember, Newsguard is only tackling a piece of the quality rater guidelines (and clearly not looking at everything Google is looking at). There are 160 pages in the guidelines and Newsguard focuses heavily on trust, credibility, and transparency.

But I was still interested in seeing the correlation. I’m not saying a failing Newsguard grade would mean you are going to tank during a major Google update, but it sure can provide some important feedback from Newsguard analysts about how your site is perceived. I plan to dig in more to see the connection between low ratings and algorithmic hits, but for now I just checked some news publishers with failing scores. I provided two examples below.

This news site failed and had previously been hammered by multiple major algorithm updates over the past two years. But, they did work hard on fixing many issues on the site recently and ended up surging during the August 1, 2018 update. So the ratings were before that happened. I’m recommending that the site contact Newsguard to provide additional information about their recent changes (and to see if they can get a fresh review).

And here’s a site that got hammered by the May 2017 update and has never come back. I haven’t analyzed this site heavily, but you can quickly see many serious issues when visiting the site. It’s pretty clear there are quality issues, UX barriers, and aggressive advertising problems throughout the site. A failing rating is justified here in my opinion, and they did get hammered by a major algorithm update.

Summary – Free quality rater feedback from Newsguard
With many marketers now heavily interested in Google’s quality rater guidelines and E-A-T, I found it interesting to learn more about Newsguard, the criteria they use to rate sites, and to see how various news publishers fared with their scoring system.

At a minimum, news publishers should review their scores and look to improve, where possible. Again, it’s like seeing what Google’s quality raters think of a site without having access to the real quality raters. I believe having a public resource like this can be extremely valuable for site owners, journalists, and SEOs. I just wish there was a Newsguard for all sites. Hey, maybe there will be some day.

GG

Google’s new index coverage reporting is killer. For a long time, the SEO community wanted something more powerful than the simple index status report in the old GSC. You know, something we could really sink our SEO teeth into. And we finally received that in the form of the new index coverage reporting in the GSC beta. I’m part of the GSC beta group and I remember how excited I was when I first started testing it out.

In the past with the index status report, you could only see very basic trending for indexation across a property in GSC. That was at least something, since site commands can be wildly off. But SEOs wanted to dig deeper, understand specific urls that were indexed, view problems with urls submitted in sitemaps, understand which urls Google was excluding from indexing (and why), and more. Well, that’s what we received with the new index coverage reporting. It was a powerful addition to the GSC beta.

Download Limits – The Bane of an SEO’s Existence
As mentioned earlier, I’m part of the GSC beta where I can help test out new functionality. I was beyond excited when testing the new index coverage report, but there were some things that needed to be improved along the way. One important item from my perspective was expanding the number of urls you can view and download from the index coverage reporting. As of now, you can only download the top one thousand urls from any given report in the index coverage tool.

I’m often working on large-scale sites with many urls (hundreds of thousands, millions, or tens of millions of urls.) As you can imagine, limiting the export number from index coverage to one thousand urls was seriously inadequate. As a quick example, here’s the crawl anomaly report with 1.2M urls. It’s important to dig into that report, but I can only view and download the top one thousand urls. Not good… not good at all.

So how can we get more data from GSC’s index coverage reporting? Is there a way to squeeze more urls out of the tool? Yes, there is, and I’m going to cover that next.

Properties in Google Search Console (GSC)
When helping new clients, it’s normal to gain access to their GSC properties as soon as I get started. But it’s all too often that I check GSC and see just one property there. For example, just the https www version of the site (or whichever is the canonical version of the site). The problem is that GSC properties are by subdomain and protocol. So, www is different than non-www, http is different than https, etc.

Therefore, at a minimum, you should have at least four properties set up in GSC for each site (http www, http non-www, https www, and https non-www). Then you can view reporting for each property and make sure you have a rounded view of your site from a GSC standpoint.

And as Forrest Gump once said, “GSC is like a box of chocolates, you never know what you’re gonna find.” OK, he never said that. But in the world of SEO, he would say that. :) There are times you might find some interesting things in non-canonical properties. Don’t let those surprise you down the line. Set them all up today. Note, you can also add each of those properties to a Property Set and view rollup reporting. I wrote about Property Sets and how to create them a few years ago and it’s smart to do.

Adding Directories As Properties in GSC: Juice Up Your Index Coverage Reporting
Four properties per site in GSC is great, but savvy SEOs have known another trick for a long time. It’s one I wrote about in an SEO galaxy far, far away (in 2014). That’s when Search Console was called Google Webmaster Tools, Matt Cutts was still at Google, medieval Panda roamed the web, and the original Penguin was busy squashing websites.

What’s the trick? Well, you can add any directory on your site as a property in GSC. And when you do, you will see reporting focused on just that directory. So, all of the reporting will only contain information for the directory you added. It’s an awesome way to get more out of GSC, including more data from any report you can export! More about that soon.

Side note about my tweet from this week:
I picked up some very interesting news this week related to GSC properties from a recent webmaster hangout video. Google’s John Mueller explained that the GSC product team is thinking about letting site owners add a root domain to GSC and then Google would verify all versions of the site in one shot. And that might include subdomains as well! This would be incredible, and the reaction from the SEO community says it all. My tweet is below, which links to the video of John explaining this:

Video from @johnmu: Whoa, Google is thinking about letting site owners add a root domain to GSC, and then Google would add *all versions* automatically (www, non-www, http, https, etc). Maybe even subdomains! That would be awesome. :) https://t.co/qTBJRHpM1m pic.twitter.com/8SmqotqALj

— Glenn Gabe (@glenngabe) June 19, 2018

Index Coverage by Directory
If you run a large-scale site, then you are probably ridiculously eager to add directories to GSC right now. But first, let me show you why it’s important for working with the new index coverage reporting. I think you’ll love this trick for getting more out of GSC (especially from a export perspective).

1. Focused reporting – Only urls from the directory you submitted
First, the various sections of the index coverage reporting will only contain urls from the directory you added. So instead of seeing millions of urls overall, you will see counts directly related to the directory you added as a property.

This can help you surface, analyze, and then fix specific problems from any given directory quicker than looking at the site overall. For example, errors, warnings, and excluded urls will only be from the directory at hand. Here’s a quick example showing the difference in urls for the site versus a specific directory:

2. Trending by directory (with notifications)
The trending you view for each report will also be focused on the directory versus the entire site. That can provide a great visual for how specific issues are trending over time (but just for the folder you are analyzing). You will see a rolling 90-day graph for each report.

And the index coverage reporting also provides notifications based on specific issues showing up. The notifications will also be focused on the directory at hand. For example, here’s a notification about urls being indexed, but blocked by robots.txt. The urls affected (and listed in the report) are only located in the directory I am analyzing. Awesome.

3. Indexation by directory
One of the great things about the new index coverage reporting is that you can finally see the urls that Google has indexed (well, at the least the top one thousand). That information is contained in the Valid category in your reporting. But now when you add a directory, you can see indexation by directory. And you can export the urls as well to get a closer look at specific pages that are valid and have been indexed. Here is an example of a directory with 858 urls indexed versus viewing many more for the entire site. And this list can be fully downloaded, since it’s under 1K urls.

4. Excluded by directory
One of the most important additions with the index coverage report was the Excluded category. This is where Google will provide urls that it’s choosing to exclude from indexing. It’s broken down by category, and you can often find glaring issues by analyzing the various reports in this section.

And now when you add directories, you can view excluded urls by directory! Again, it’s a focused set of reports versus lumping together urls from across the entire site. For example, one of my favorite reports is “Google chose a different canonical than user”. That’s where you can view urls that contain a canonical tag, but Google chose to ignore that tag and index a different url. As you can imagine, it’s important to understand what’s gong on there. And that’s especially the case if you see thousands of urls in the report (or more). Below, the directory has 405 urls versus many more at the site-level. Again, this helps greatly when analyzing a problematic situation.

5. Downloads by directory
This is one of the most important benefits you receive by adding directories in GSC. The index coverage reporting enables you to download any report, but it’s limited to the top one thousand urls per report. For smaller sites, that number might suffice, but for larger-scale sites, it’s typically very limiting.

But when you add directories, the reporting will not contain urls from across the entire site. It will only contain urls for the directory you are analyzing. Therefore, you can export more urls for the directory at hand. That includes urls that are indexed, as well as, urls that are being excluded from indexing. It’s a more surgical way to export urls.

For example, when checking the https www property for this site, you can see 312K soft 404s. But when checking just the directory I added, you can see 4K. So, when you export the top one thousand from the directory property, you’ll be getting more data specifically for that folder (1/4 of the total). That’s versus trying to export one thousand out of 312K. And then you can do this for other important directories to see where the core problems are per directory.

Summary – Get all of the data you can by adding directories in GSC
The new index coverage reporting in GSC is powerful, and it was a huge improvement from the index status report in the old GSC. That said, the reporting will only list the top one thousand urls per category, which can be very limiting for larger-scale sites. But if you add directories, you can focus the reporting on just that folder (versus viewing urls from across the entire site). And when you do, your downloads will be more focused as well (which can get you even more data for the problem at hand). So go ahead and add your key directories today. Data awaits. :)

GG