Google’s new index coverage reporting is killer. For a long time, the SEO community wanted something more powerful than the simple index status report in the old GSC. You know, something we could really sink our SEO teeth into. And we finally received that in the form of the new index coverage reporting in the GSC beta. I’m part of the GSC beta group and I remember how excited I was when I first started testing it out.

In the past with the index status report, you could only see very basic trending for indexation across a property in GSC. That was at least something, since site commands can be wildly off. But SEOs wanted to dig deeper, understand specific urls that were indexed, view problems with urls submitted in sitemaps, understand which urls Google was excluding from indexing (and why), and more. Well, that’s what we received with the new index coverage reporting. It was a powerful addition to the GSC beta.

Download Limits – The Bane of an SEO’s Existence
As mentioned earlier, I’m part of the GSC beta where I can help test out new functionality. I was beyond excited when testing the new index coverage report, but there were some things that needed to be improved along the way. One important item from my perspective was expanding the number of urls you can view and download from the index coverage reporting. As of now, you can only download the top one thousand urls from any given report in the index coverage tool.

I’m often working on large-scale sites with many urls (hundreds of thousands, millions, or tens of millions of urls.) As you can imagine, limiting the export number from index coverage to one thousand urls was seriously inadequate. As a quick example, here’s the crawl anomaly report with 1.2M urls. It’s important to dig into that report, but I can only view and download the top one thousand urls. Not good… not good at all.

So how can we get more data from GSC’s index coverage reporting? Is there a way to squeeze more urls out of the tool? Yes, there is, and I’m going to cover that next.

Properties in Google Search Console (GSC)
When helping new clients, it’s normal to gain access to their GSC properties as soon as I get started. But it’s all too often that I check GSC and see just one property there. For example, just the https www version of the site (or whichever is the canonical version of the site). The problem is that GSC properties are by subdomain and protocol. So, www is different than non-www, http is different than https, etc.

Therefore, at a minimum, you should have at least four properties set up in GSC for each site (http www, http non-www, https www, and https non-www). Then you can view reporting for each property and make sure you have a rounded view of your site from a GSC standpoint.

And as Forrest Gump once said, “GSC is like a box of chocolates, you never know what you’re gonna find.” OK, he never said that. But in the world of SEO, he would say that. :) There are times you might find some interesting things in non-canonical properties. Don’t let those surprise you down the line. Set them all up today. Note, you can also add each of those properties to a Property Set and view rollup reporting. I wrote about Property Sets and how to create them a few years ago and it’s smart to do.

Adding Directories As Properties in GSC: Juice Up Your Index Coverage Reporting
Four properties per site in GSC is great, but savvy SEOs have known another trick for a long time. It’s one I wrote about in an SEO galaxy far, far away (in 2014). That’s when Search Console was called Google Webmaster Tools, Matt Cutts was still at Google, medieval Panda roamed the web, and the original Penguin was busy squashing websites.

What’s the trick? Well, you can add any directory on your site as a property in GSC. And when you do, you will see reporting focused on just that directory. So, all of the reporting will only contain information for the directory you added. It’s an awesome way to get more out of GSC, including more data from any report you can export! More about that soon.

Side note about my tweet from this week:
I picked up some very interesting news this week related to GSC properties from a recent webmaster hangout video. Google’s John Mueller explained that the GSC product team is thinking about letting site owners add a root domain to GSC and then Google would verify all versions of the site in one shot. And that might include subdomains as well! This would be incredible, and the reaction from the SEO community says it all. My tweet is below, which links to the video of John explaining this:

Video from @johnmu: Whoa, Google is thinking about letting site owners add a root domain to GSC, and then Google would add *all versions* automatically (www, non-www, http, https, etc). Maybe even subdomains! That would be awesome. :) https://t.co/qTBJRHpM1m pic.twitter.com/8SmqotqALj

— Glenn Gabe (@glenngabe) June 19, 2018

Index Coverage by Directory
If you run a large-scale site, then you are probably ridiculously eager to add directories to GSC right now. But first, let me show you why it’s important for working with the new index coverage reporting. I think you’ll love this trick for getting more out of GSC (especially from a export perspective).

1. Focused reporting – Only urls from the directory you submitted
First, the various sections of the index coverage reporting will only contain urls from the directory you added. So instead of seeing millions of urls overall, you will see counts directly related to the directory you added as a property.

This can help you surface, analyze, and then fix specific problems from any given directory quicker than looking at the site overall. For example, errors, warnings, and excluded urls will only be from the directory at hand. Here’s a quick example showing the difference in urls for the site versus a specific directory:

2. Trending by directory (with notifications)
The trending you view for each report will also be focused on the directory versus the entire site. That can provide a great visual for how specific issues are trending over time (but just for the folder you are analyzing). You will see a rolling 90-day graph for each report.

And the index coverage reporting also provides notifications based on specific issues showing up. The notifications will also be focused on the directory at hand. For example, here’s a notification about urls being indexed, but blocked by robots.txt. The urls affected (and listed in the report) are only located in the directory I am analyzing. Awesome.

3. Indexation by directory
One of the great things about the new index coverage reporting is that you can finally see the urls that Google has indexed (well, at the least the top one thousand). That information is contained in the Valid category in your reporting. But now when you add a directory, you can see indexation by directory. And you can export the urls as well to get a closer look at specific pages that are valid and have been indexed. Here is an example of a directory with 858 urls indexed versus viewing many more for the entire site. And this list can be fully downloaded, since it’s under 1K urls.

4. Excluded by directory
One of the most important additions with the index coverage report was the Excluded category. This is where Google will provide urls that it’s choosing to exclude from indexing. It’s broken down by category, and you can often find glaring issues by analyzing the various reports in this section.

And now when you add directories, you can view excluded urls by directory! Again, it’s a focused set of reports versus lumping together urls from across the entire site. For example, one of my favorite reports is “Google chose a different canonical than user”. That’s where you can view urls that contain a canonical tag, but Google chose to ignore that tag and index a different url. As you can imagine, it’s important to understand what’s gong on there. And that’s especially the case if you see thousands of urls in the report (or more). Below, the directory has 405 urls versus many more at the site-level. Again, this helps greatly when analyzing a problematic situation.

5. Downloads by directory
This is one of the most important benefits you receive by adding directories in GSC. The index coverage reporting enables you to download any report, but it’s limited to the top one thousand urls per report. For smaller sites, that number might suffice, but for larger-scale sites, it’s typically very limiting.

But when you add directories, the reporting will not contain urls from across the entire site. It will only contain urls for the directory you are analyzing. Therefore, you can export more urls for the directory at hand. That includes urls that are indexed, as well as, urls that are being excluded from indexing. It’s a more surgical way to export urls.

For example, when checking the https www property for this site, you can see 312K soft 404s. But when checking just the directory I added, you can see 4K. So, when you export the top one thousand from the directory property, you’ll be getting more data specifically for that folder (1/4 of the total). That’s versus trying to export one thousand out of 312K. And then you can do this for other important directories to see where the core problems are per directory.

Summary – Get all of the data you can by adding directories in GSC
The new index coverage reporting in GSC is powerful, and it was a huge improvement from the index status report in the old GSC. That said, the reporting will only list the top one thousand urls per category, which can be very limiting for larger-scale sites. But if you add directories, you can focus the reporting on just that folder (versus viewing urls from across the entire site). And when you do, your downloads will be more focused as well (which can get you even more data for the problem at hand). So go ahead and add your key directories today. Data awaits. :)

GG

I’ve written about Murphy’s Law for SEO before, and it’s scary as heck. And that’s especially the case for large-scale websites with many moving parts. Murphy’s law is an old adage that says, “anything that can go wrong, will go wrong.” For example, no matter how much you plan and prep for large-scale SEO changes, there’s a good possibility that something will go wrong.

And when SEOs are implementing major changes across complex sites, whether that be CMS migrations, domain name changes, https migrations, etc., you can bet Murphy will be hanging around. You just need to be prepared for Murphy to pay a visit.

I recently helped a company move to https, and overall, it was a clean migration. I’ve helped this company with a number of SEO projects over the past several years and I was extremely familiar with their site, content, and technical SEO setup. The site was doing extremely well SEO-wise leading up to the migration, so we definitely wanted to make sure the move to https went as smoothly as possible.

We prepared heavily leading up to the migration, so we could all be confident when finally pulling the trigger. When a site moves to https there are a number of checks I perform prior to, during, and then after the migration takes place. And that includes several crawls to ensure everything is ok.  In addition, my client is SEO savvy and they were checking the site from their end as well.

But remember our friend Murphy? Read on.

Site Commands Are Useless, Or Are They?
As we were tracking some of the normal volatility you can see during a migration, my client was checking indexation across some important urls. While performing some site commands, they noticed a rogue subdomain showing up with copies of the core website’s content. When I quickly checked that subdomain (via a site command) the subdomain revealed close to 7K pages indexed.

It seems when they pulled the trigger and moved to https, Murphy came to pay a visit. A subdomain that was never existent, suddenly appeared. And for some reason, the site was duplicating all pages from the core site on that subdomain.

Luckily, we were relatively early into the migration so we were able to start tackling the problem quickly. We clearly wanted to get those urls out of Google’s index as quickly as possible and then handle the subdomain properly. Below, I’ll document what we did to overcome a rogue subdomain showing up containing a lot of duplicate content.

Indexation, Rankings, and Traffic
One of the first things I told my client was to add that rogue subdomain to Google Search Console (GSC). That would enable us to see reporting directly from Google for that subdomain, including indexation, search analytics data, crawl errors, etc. It would also enable us to submit xml sitemaps and use the Removal Urls Tool if we thought that would be helpful. They quickly set up the GSC property and we waited for data to come in.

While we waited for data to arrive in GSC, I told my client to make sure all of the urls across the rogue subdomain 404. They did that pretty quickly. So the urls were only active for a relatively short period of time, but long enough for thousands to be indexed.

Once data arrived in GSC for the subdomain, we were able to see exactly what we were dealing with. There were 7,479 urls indexed according to Index Status reporting in GSC.

In addition, some of the urls were actually ranking for queries and driving clicks. We saw 759 clicks and 28,724 impressions once the urls were indexed. Note, that was a small percentage of Google organic traffic compared to the core website, but clearly not a good thing that the rogue subdomain was driving impressions and clicks.

So we had a problem on our hands. Below, I’ll document the steps we took to remove the rogue subdomain from the Google search results while also working to have the pages removed from the index long-term. We also wanted to make sure that any subdomain urls that were ranking in the SERPs were correctly switched to the core site urls.

How To Remove Rogue URLs From Google’s Index:

1. 404 (or 410) all of the pages.
I mentioned this earlier, but I told my client to make sure all urls on the subdomain were 404ing. And yes, you can use 410s if you want. That could quicken up the process slightly for having the urls removed from the index. John Mueller has explained in the past that it could quicken up the process “a tiny bit”.
Here’s a video of John explaining that:

2. Verify the property in Google Search Console (GSC) – for several reasons.
We verified the subdomain in GSC so we could gain reporting directly from Google related to indexation, search traffic and impressions, xml sitemaps, crawl errors, and more. In addition, we would have access to the Remove Urls Tool, which can be very helpful with quickly removing urls from the Google search results (temporarily).

3. Create and submit an xml sitemap with all urls that are 404ing (via GSC).
We created an xml sitemap with all rogue urls and submitted that sitemap via GSC for the subdomain in question. That can help Google find the rogue urls that are now 404ing and hopefully get them out of the index quicker.

4. Use the Removal URLs Tool in GSC for the new property.
In Google Search Console (GSC), there is a very powerful, yet dangerous, tool called the Remove URLs Tool. I find there’s a lot of confusion about how to use the tool, how it works, what will happen long-term, etc.

First, it’s important to know that it’s a temporary removal (90 days). Using the tool, you can temporarily remove urls from the Google SERPs (in case the urls contain sensitive information, confidential information, etc. that shouldn’t be displayed in the search results). This is probably the most common misconception. Some site owners remove a url or a directory using the tool and then think it’s gone forever. That’s not the case. You have to make sure you remove the urls via 404 or 410, or by noindexing the urls, in order for them to be truly removed from the index. You can also put the content behind a login (forcing a username/password to log in). If not, the urls can reappear 90 days later.

Second, you can remove specific urls, directories, or entire sites by using the tool in GSC. And the Remove URLs Tool is case-sensitive and precise (character by character). So make sure you enter the exact url when using the tool, or it might not work the way you want it to. Also, to remove an entire site or subdomain, just leave the form blank and click the button labeled “Continue”. All of the urls for the root GSC property you are dealing with will be removed.

Since we were dealing with a rogue subdomain, we wanted to nuke all of the urls residing there. So we simply left the field blank for the subdomain in GSC and pushed the giant nuclear button. Below you can see the subdomain removal request based on using the Remove URLs Tool.

Important Note: Make sure you are working with the right GSC property or you can incorrectly remove your core website from the SERPs. Remember I said the tool can be dangerous?

How Fast Will The URLs Be Removed?
That’s a trick question. There are two ways to think about the problem. First, how fast will the urls be removed from the SERPs? And then how fast before Google truly deindexes the urls (for the long-term). The Remove URLs Tool can work very quickly. Google’s John Mueller has said it can take less than a day for the urls to be removed from the SERPs, but I’ve seen it work even faster.

For this situation, it was just a few hours before a site command showed 0 results.

But remember, it’s a temporary removal. You still need to handle the urls on the site correctly in order for them to be removed over the long-term. That means having the urls resolve with a 404 or 410, putting the content behind a login, or by using the meta robots tag using “noindex”. If you do one of those things, then the urls won’t reappear in Google’s index 90 days later.

Bonus: Double Check Logs To Ensure The Pages Aren’t Being Visited Anymore
And for those of you that dig analyzing log files, you can check your logs to make sure traffic drops off to the urls on the rogue subdomain. You can use Screaming Frog Log File Analyzer to import your logs and check activity (and hopefully see declining activity.)

Moving Forward: Monitoring The Problem Over The Long-Term
Once the above steps are completed, you shouldn’t just sit back with a frozen margarita in your hand gazing at Google Analytics. Instead, you should monitor the situation to make sure all is ok long-term. That includes reviewing the GSC reporting for the rogue subdomain, which can reveal several key things.

1. Make sure indexation drops.
Check the index status report and perform site commands to make sure the urls remain out of Google’s index. If you see them come back for some reason, make sure the urls are 404ing, they are properly noindexed, or behind a login. You should also make sure someone didn’t reinclude the urls in the Removal URLs Tool in GSC.

2. Make sure 404s show up in crawl errors.
By checking the crawl errors reporting in GSC, make sure the urls are showing up as 404s. For the subdomain we were working on, you can see the rise of 404s since my client implemented the correct changes. If you don’t see 404s increasing, you might not be correctly implementing 404s across the site or subdomain.

3. Make sure traffic drops off.
By checking the search analytics reporting in GSC, you can see impressions and clicks over time. Make sure they are dropping after removing the urls from Google. If you don’t see a drop in impressions and clicks for the rogue urls, then double check the Remove URLs tool, make sure the urls are indeed 404ing, being noindexed, etc.

Summary – Removing URLs… and Showing Murphy The Exit Door
If you find yourself in a situation where a rogue subdomain gets indexed with duplicate content, then use all of the tools available to you to rectify the situation. And that includes the Remove URLs Tool in GSC. Unfortunately, when implementing large-scale changes on complex sites, site owners should understand that things can, and will, go wrong. Murphy’s Law for SEO is real and can definitely throw a wrench into your operation. My advice is to be ready for Murphy to show up, and then move quickly to kick him out.

GG