The Internet Marketing Driver

Smart Delta Reports – How To Automate Exporting, Filtering, and Comparing Google Search Data Across Timeframes Via The Search Console API and Analytics Edge

By Leave a Comment

How to automate a delta report via Analytics Edge in Excel.

In 2013, I wrote a post explaining how to create what I called a Panda Report, which enabled you to identify landing pages seeing the biggest drop during a major algorithm update. The post explained how to do this based on Google Analytics data, but you can definitely do the same thing with GSC data (and with queries in addition to landing pages).

Well, it’s 2021, the process I use has been enhanced, and I wanted to publish a new post explaining how to automate the process using Analytics Edge. First, since medieval Panda is now missing in the SEO trees, the report needed a new name. For the past several years, I’ve simply called it a Delta Report. That fits much better since you can use this approach to identify the change in impressions, clicks, or rankings based on any event impacting Search (like a broad core update, a site migration, website redesign, or any other situation causing volatility). By identifying the landing pages and/or queries seeing the biggest drop, you can often find glaring issues. It’s a great way to start digging into the data after a big drop or surge in rankings and traffic.

And beyond the name change, there are some great ways to bulk export data via the Search Console API now. Since the GSC UI limits exports to just one thousand rows of data per report, using the Search Console API is critically important for exporting all of your data. For example, I often use the API to mass-export landing pages and queries from GSC (going well beyond the one thousand row limit).

A Delta Report is a Panda Report on Steroids – The Power of Analytics Edge and APIs
I have covered Analytics Edge a number of times before in blog posts about exporting data. It’s an add-on for Microsoft Excel that quickly and efficiently enables you to bulk export data via the Search Console API, Google Analytics API, and more. But that’s not all you can do with Analytics Edge. Beyond just the export, you can create macros that filter and organize the data to create advanced reports (so you can export, compare, filter, etc. and all in one shot).

For our purposes, that means exporting all queries or landing pages, comparing the data based on timeframe, filtering based on your site structure, important query types, etc., and then writing those exports to their own worksheets for further analysis. And this is all done in one shot based on the macro you create (build once, use often).

Excited? Below, I’ll walk you through the process of creating the macro via Analytics Edge. And while you go through this tutorial, you’ll probably think of 100 other things you can use Analytics Edge for while working with data. It’s basically a Swiss Army Knife for working with APIs.

How to export GSC data via the Search Console API, compare data across timeframes, filter the data by page or query type, and create separate worksheets, all in one shot:
First, there are some requirements. Although Analytics Edge isn’t free, it’s extremely cost-effective. The core add-in costs $99 per year and the GSC connector costs $50 per year. Also, the good news is that there is a 30-day free trial so you can walk through this tutorial and use the process for 30 days to see how it works for you.

So, for $150 per year, you can use Analytics Edge to your heart’s delight. If you are helping larger sites where the API is necessary to export all of your data, then Analytics Edge is a great way to go, and it definitely won’t break the bank.

How To Create A Delta Report: Step-by-step instructions

1. Download and Install Analytics Edge (Core Add-in):
You can download and run the Analytics Edge installer to quickly install the add-in. After installing the add-in, click the license button and accept the Terms of Use. Once you do, a 30-day free trial will start for the core add-in.

2. Install the GSC Connector from within Excel:
Now that the Core Add-in is installed, you need to add the GSC connector so you can work with the Search Console API. It’s very easy to install the connectors available in Analytics Edge. Simply click the License button in the Analytics Edge menu and then click the dropdown to add a new connector. Select Google Search and then click Add. Then click Install. Once the connector is installed, a 30-day trial will begin for that connector.

3. Connect Your Google Account:
In order to export data from your GSC properties, you first need to connect your Google Account associated with those properties. Once you set this up, you will not need to do this over and over. And you can add multiple Google Accounts if you have access to GSC properties across various accounts. Click the Google Search dropdown in the Analytics Edge menu and select Accounts. Click Add Account and walk through the process of quickly connecting your Google account. Once you do, you’ll be able to use the API to connect to any property you have access to.

4. Create Your Macro – Exporting All Landing Pages or Queries and Comparing Data Across Timeframes:
Let’s start creating our macro by exporting all landing pages and comparing data across timeframes. In a real-world situation, you would compare the timeframe after a major event (like a broad core update) to the previous timeframe to see the changes per landing page or query. But for this tutorial, we’ll keep it simple. Let’s just pull all landing pages for the last 28 days and compare to the previous timeframe. Once you get the hang of this, you can customize the report for each situation you encounter. Note, if you ever lose the Macro window, just click “Task Pane” in the Analytics Edge menu in Excel. It will show back up on the right side of the spreadsheet. Let’s start creating our macro. Go ahead and click Analytics Edge in the main menu, then Google Search, and then Search Analytics. Name your macro DeltaReport (or whatever you want).

5. Choose The Account and GSC property:
Select the Google account you want to use and then select the GSC property in the site list.

6. Select Fields To Export:
In the available dimensions and metrics list, select page and click “Add” to export all landing pages from Google organic Web Search. Leave the selected metrics as-is (with clicks, impressions, ctr, and position all selected).

7. Leave Filters Tab As-Is, But Review The Settings:
For this tutorial, we’ll leave the filters tab as-is, but note the options you have here while exporting data from GSC. You can filter by page, query, country, device type, search type, and search appearance. You’ll notice the default search type is Web Search. That’s what we want for this specific report, so keep the default settings.

8. Select A Date Range – Comparing Data Across Timeframes Made Easy:
Depending on your situation, select the appropriate timeframe for exporting data. For this tutorial, let’s pull the last 28 days of data and compare to the previous timeframe. Click the Dates tab and simply use the dropdown to select Last 28 Days. To compare timeframes, make sure to select the checkbox for “Compare to” and select a timeframe to compare with. To keep things simple, we’ll just select “Previous period”.

9. Sort By Clicks Or Impressions:
Under the Sort/Count tab, use the dropdown to select a metric to sort the data by. I typically choose clicks in descending order. Make sure to click the button labeled “Descending” to apply the sort preference.

10. Run The Query!
Click OK in the bottom right corner of the wizard to run the query. Depending on how much data needs to be exported, it can take a few seconds (or longer). Once the query has completed, you will see the results highlighted in green. Note, this does NOT show the full results from the export. Analytics Edge just shows a sample of the results and is waiting for more input from you (either to write the full data to a worksheet now or to use the built-in functions to create more advanced reports).

11. Set The Table Name:
Since we’ll be filtering the data we just exported multiple times (by page type), we need to set the table name so we can come back to the full data in future steps. To do this, click the Table dropdown in the Analytics Edge menu and select Table Name. Set the table name to something like “allpages” and click OK. Again, we’ll need this in the future.

12. Write To Worksheet:
Let’s complete the export by writing the data to a new worksheet. In order to do this, click the File dropdown in the Analytics Edge menu and then select Write to Worksheet. Name your worksheet something like “Landing Pages All Data” and then click OK. Analytics Edge will create a new worksheet containing the full export from GSC. Just click the new worksheet to view all of the data. You’ll notice all of the landing pages were exported with columns showing the difference in clicks, impressions, CTR, and position based on comparing the last 28 days to the previous timeframe. Awesome, right? But we’re not done yet. Our macro will be smarter than that. :)

13. Start Filtering Your Data:
Our goal is to create separate worksheets by page type so you can easily analyze each one separately. To keep things simple, let’s say we wanted to break out category pages (/category/), product pages (/products/), and blog posts (/blog/) so we could analyze them separately. Let’s start with category pages. Click the Table dropdown in the Analytics Edge menu and select “Filter”. This menu will enable you to filter the data by any column in the active table. The active table now is “allpages”, which we set up in step 11. Once you click “Filter in the menu”, you can set the filter rules. The column should say “A page”, which will enable you to filter by the column in our active table titled “page”. For Criteria, you have several helpful options, including regex. Yes, you can use regex here if needed, which is awesome. To filter by category pages which contain /category/ in the url, select “Contains” and then enter /category/ in the Value text box. Then click the Add button. Note, you can combine rules here if you want to create more complex filtering options. Click OK to filter the active table. You will see the results again highlighted in green. We’ll write the filtered data to a worksheet in the next step.

14. Write The Filtered Data To A Worksheet:
Just like we did before, let’s write the filtered category data to a new worksheet. Click the File dropdown in the Analytics Edge menu and select “Write Worksheet”. Then name the new worksheet “Landing Pages Category” and click OK. A new worksheet will be created with all of the category page data. At this point, you should have two worksheets, one containing all landing page data and another just containing category page data.

15. Switch The Table Back In Order To Filter Again:
Now we want to filter the full data again for product pages, which contain /product/ in the url. In order to do that, we need to switch the active table back to “allpages”, which contains all of our exported data and not just the filtered category data. If we don’t switch the table name again, then Analytics Edge will use the current active table, which is the category page data. In order to switch the table, click the Table dropdown in the Analytics Edge menu and click Table Name. Click the second radio button to switch to a previously named table and select “allpages”. Now that becomes the active table and we can filter it again.

16. Filter Product Pages:
We’ll use the same approach that we did when filtering the category pages, but this time, we’ll filter by urls containing /products/. After switching the table name, select Table again in the Analytics Edge menu and then Filter. Now enter /products/ in the value field for page. Then click OK. The data will now be filtered by any url with /products/ in it.

17. Write Filtered Data To Worksheet:
Next, we need to write the product filtered data to a new worksheet. Click the File dropdown in the Analytics Edge menu and select “Write Worksheet”. Name the worksheet “Landing Pages Products” and click OK. You will now have a new worksheet with the filtered data. And now you should have three worksheets containing landing page data (full data, category pages, and product pages).

18. Rinse and Repeat For Blog URLs:
Use the same approach to export all data filtered by blog urls (containing /blog/ in the url). First, switch the table name back to “allpages” (see step 14 for how to do this). Then filter the data by any url containing /blog/, and then write to a new worksheet called “Landing Pages Blog”. When you’re done, you should have four worksheets in total with three that contain filtered data (one for category urls, one for product urls, and one for blog urls). And Analytics Edge already took care of comparing data across timeframes and provided difference columns in the worksheets.

Congratulations! You have exported all of your landing pages from GSC, compared data across timeframes, filtered by page type, and then created specific worksheets containing the filtered data. Oh, and now you have a macro using Analytics Edge that you can reuse whenever you want to accomplish a similar task in the future. Just reopen the spreadsheet, save a new file, edit the macro to change the settings like GSC property, click “Refresh All” in the upper left corner of the Analytics Edge menu, and boom, you’re good to go. Time is valuable and this can save you a lot of it in the future…

Beyond The Delta Report: More Functionality = More Advanced Reporting
As I mentioned earlier, Analytics Edge comes with a ton of functionality built-in. You can create advanced reporting by using the various functions available in Analytics Edge when working with data exported from GSC, Google Analytics, and more. So, if you’re feeling ambitious, here are some other things you can try using Analytics Edge and the GSC Connector:

  • Run the same type of report, but for queries instead of landing pages. Then you can analyze drops or surges by query type instead of page type.
  • You can segment by search type to analyze drops and surges for Image Search, Video Search, or the News tab.
  • You can segment by search feature (like AMP, how-to, FAQs, Q&A, reviews, recipes, etc.) Note, you can follow my tutorial for exporting data by search appearance to learn more about that process.
  • And make sure to review all of the functions available in the Analytics Edge menu within the Multiple, Table, and Column dropdowns. For example, you can filter, group, pivot, sort, append, combine, compare, convert, split, and more. Again, Analytics Edge is like a Swiss Army Knife for APIs.

Summary – Automated Delta Reports are Panda Reports on Steroids
It’s always smart to analyze the top landing pages and/or queries when a site sees a big drop or surge in rankings and traffic from Google (due to an algorithm update, site migration, website redesign, etc.) Since the Performance reporting in the GSC UI limits exports by 1K rows, using a tool like Analytics Edge can help you quickly and efficiently export all of your data via the Search Console API.

In addition, Analytics Edge comes with a number of functions for filtering and working with your data to create advanced reports (including comparing data by timeframe). By following this tutorial, you can create a template for quickly exporting data, comparing data across timeframes, filtering by page or query type, and then writing the results to separate worksheets for further analysis. Once you get the hang of Analytics Edge, the sky’s the limit. I think you’ll dig it.

GG

Filters and Pills in the Google SERPs – How the addition of filters, tabs, and dynamic organization in the search results can impact visibility and clicks

By Leave a Comment

Filters and pills in the Google search results

As Bob Dylan famously wrote, “The times, they are-a changin’…” Well, based on Google’s frantic pace of change, SEOs could easily tweak that line and replace “times” with “SERPs”. In this post, I’ll cover two more changes that site owners and SEOs should be aware of. And those additions can have a big impact on visibility and clicks when they appear in the search results.

Over the past few years, I’ve written several posts about the power of certain SERP features and how those features can impact a site’s visibility and traffic. For example, I wrote a post covering Interesting Finds, which is a super-powerful SERP feature that takes up a huge amount of real estate in the mobile search results. And then I also wrote about Found on the web, which is a Frankenstein-like feature that can also take up massive real estate while yielding multiple carousels of listings.

Both of those features are important to understand, but in true Google form, there are more changes I wanted to bring to your attention today. And although these might be slightly more subtle than Interesting Finds and Found on the web, they can still pack a lot of punch.

Below, I’ll cover what the addition of filters and pills in the search results means for your business. During my research, I have seen them show up more and more recently for certain types of queries. For example, I’ve seen them show up for how-to queries and for queries that affiliate marketers might be targeting (like a search for “best {product category}” or even just the product category in general). Filters and pills don’t impact all sites and queries, but in my opinion, they are important for all site owners to at least be aware of.

Filters and Pills – Not a new punk band, but they can sure rock a SERP
I’ll start with filters, which have rolled out for specific types of queries and can be seen on both desktop and mobile. When they trigger, you will see a set of filters at the top of the search results that are related to the query you searched for. On desktop, they show up with a small magnifying glass icon, like this:

filters in the Google desktop search results

And on mobile, they show up as rectangular facets at the top of the search results:

Filters in the Google mobile search results.

Similar to e-commerce functionality, they enable users to filter the search results with the click of a button. On that note, it’s important to know that filters change the initial query to reflect the filter you selected.

For example, if you search for “best hair treatment”, you will see filters show up containing keywords related to hair treatment. If you then click the filter for “hair growth”, a new query will be triggered for “the best hair treatment for hair growth” and the search results changes.

Filter in the Google search results

After clicking or tapping a filter, the other filters still remain at the top of the search results, enabling users to continue to filter based on additional types of hair treatment. Again, very e-commerce functionality-like.

And here is what it looks like on mobile after initially searching for “best hair treatments” and then tapping the filter for “frizzy”:

Filter in the Google mobile search results

New queries, new opportunities:
The most obvious takeaway here is that sites that rank for the initial query could see people utilizing the filters and being taken to completely new search results. If that’s the case, and those site owners don’t have content that can rank for those additional queries, then they could be losing visibility and traffic. Again, the filters trigger a completely new search, yield completely new listings, and even new SERP features in some cases.

For example, you might find new SERPs with Interesting Finds, videos with key moments, “found on the web”, and more. So, while using those filters could send users to completely new SERPs, it also provides completely new opportunities for site owners. But, you need to be aware this is happening in the first place, which is why analyzing the actual SERPs is incredibly important. I’ll cover more about that soon. Next up, I’ll cover pills.

Dynamic organization and pills:
Google has been expanding what it calls dynamic organization for years. That’s where Google is organizing subtopics in the SERPs related to the core entity or topic you have searched for. You will often see dynamic organization in action for various entities in the SERPs. The data is organized into tabs and users can click each tab to view additional information related to that entity or topic. Tabs of the past have become colorful pills based on Google’s mobile redesign, which rolled out in January of 2021.

First, I’ll cover dynamic organization overall and then I’ll cover a newish version that I’ve been seeing more and more recently. Bear with me, we’ll get to the good part soon.

If you do a search for a music band, type of dog, movie, or even a topic like machine learning and you can often see a knowledge panel employing dynamic organization. In addition, on desktop you will sometimes even see a left-side navigation with tabs containing subtopics. And on mobile, those tabs are presented as pills (or small rounded buttons).

For example, here is a search for AC DC. The knowledge panel shows up with subtopics (pills):

Dynamic organization in the Google search results for AC/DC.

And here is a search for the movie Escape From New York:

Dynamic organization in the Google search results for Escape from New York.

But this is happening beyond traditional entities most people think of. Here is a search for machine learning that triggers subtopics (pills):

Dynamic organization in the Google search results for machine learning.

Dynamic organization has expanded over the years to cover entertainment, weather, music, travel, sports, and other categories. For example, you can often see dynamic organization in action for actors, movies, books, artists, and many other types of entities and topics. The mobile version was implemented in 2018 by Google, while the desktop left-side navigation has been expanding more recently. Also, you will often see tests being run as Google continually experiments with the best way to present the data.

Here is another example, but this time for the entity “unit circle”, which is a mathematical concept. Yes, we are way beyond the typical entities that most people think of…

Dynamic organization in the Google search results for unit circle.
Dynamic organization in the Google search results for unit circle in the mobile search results.

Most SEOs will be familiar with what I have shown so far, but there’s a newish version of dynamic organization that has appeared more and more recently. I’ll cover that next.

Just the pills: An expansion of dynamic organization
What I’m actually focusing on today isn’t the typical knowledge panel-like feature I mentioned above. Instead, Google has started showing just the pills for certain queries and not the full knowledge panel-like result. And it’s for many queries where people are searching for products or services (so this implementation of dynamic organization is impacting commerce).

For example, if you search for “best mixers” on mobile, you will see several pills at the top of the SERPs for “overview”, “products”, “videos”, “stores”, and “reviews”.

You can see the initial SERP below:

Just pills in the Google mobile search results.

For site owners, it’s important to understand that the pills (or tabs) do not trigger a new search, but instead, dynamically load different results in the SERPs. This is similar behavior you might see on an e-commerce site that uses AJAX to load or filter results without having to reload the entire page.

For example, tap “reviews” and this is what you see:

Review tab in the Google search results on mobile.

And remember, the initial query remains intact, but the listings are changing. So, similar to what I explained earlier, if users search for a query and you rank in the top 10, but then a user taps a pill to learn more about a subcategory (like reviews), then new listings will show up in the SERPs. And if you don’t have content that can rank for those subcategories, you could be losing visibility and traffic.

Here is another example for the query “Bluetooth speakers”. Here is the initial results with pills appearing at the top:

Tabs in the mobile search results.

But when you tap the “videos” pill, the results completely change to all videos:

Video tab in the mobile search results.

From a tracking perspective, you could theoretically see impressions remain stable as your listings rank in the default top ten, but clicks might drop as some people utilize the pills to research other subtopics related to that entity or topic.

And on the flip side, you might see impressions and clicks from queries where you don’t rank in the default top 10. Instead, users might be tapping a pill to filter the results and that’s where your listing shows up. In GSC, those impressions would register once the listings show up in the SERPs (after a user taps the tab).

Here is what it looks like in practice. Notice how the SERP is changing in the gif below, but the initial query remains intact:

Basically, Google is playing chess with this SERP… and I would make sure you’re not playing checkers. :)

How do you know when filters and pills are impacting your top queries?
Now that I’ve introduced filters and pills in the SERPs, I’m sure you are wondering how to track this for your own site to understand if rankings are being impacted. The quick answer is you won’t (easily) know. You have to find examples of this happening on your own. As you can guess, you can’t filter by these features in GSC, so you have to come up with your own process for surfacing queries that are impacted.

If you think you could be impacted by filters or pills based on what I provided earlier in this post, I recommend bulk exporting your queries from GSC via the API (using Analytics Edge, Data Studio, etc.) Then check queries that might be declining in clicks, but have stable impressions. That could surface a situation where your pages are ranking in the default top 10, but users are clicking the filters or tabs, which load new results.

If you see stable impressions, but declining clicks, it could be due to certain SERP features at play (like filters and pills). I would check the SERPs manually for those queries to see if filters or pills are showing up.

Gap Analysis: What can you do if filters or pills are present for your important queries?
If you find filters or pills are triggering for important queries, then go through the process of checking the filtered results to understand the pages ranking there and then perform a gap analysis with your own content.

Is your site’s content only present in the default listings or do you have multiple pieces of content that can rank when the SERPs are filtered? If there are gaps, close those gaps. There’s no guarantee you can recover that visibility, but you will have no shot if you don’t have content that matches user intent based on those filters.

SERP Features Can Have A Big Impact, Especially On Mobile
I’ve always said that certain SERP features can have a big impact on visibility and traffic from the search results. And that’s especially the case on mobile where you don’t have much room to play with. With filters and pills, the entire SERP is changing based on the tap of a button. And if your listings aren’t there when the SERPs are filtered, you could be losing visibility and clicks from Google. But, it also presents an opportunity for your content team to perform a gap analysis and close those gaps.

Filters and pills obviously aren’t impacting every query, but they can have a big impact for the queries where they trigger. If you are producing content focused on the types of queries I mentioned earlier in the post, I would take a hard look at the actual SERPs to see which features are present. You never know, you might see filters and/or pills showing up. If that’s the case, I would form a plan of attack for dealing with them.

GG

How To Use GSC’s Crawl Stats Reporting To Analyze and Troubleshoot Site Moves (Domain Name Changes and URL Migrations)

By Leave a Comment

For site migrations, I’ve always said that Murphy’s Law is real. “Anything that can go wrong, will go wrong.” You can prepare like crazy, think you have everything nailed down, only to see a migration go sideways once it launches.

That’s also why I believe that when something does go wrong (which it will), it’s more super-important to address those problems quickly and efficiently. If you can nip migration problems in the bud, you can avoid those problems becoming major issues that impact SEO. That’s why it’s important to prepare as much as you can, have all of the necessary intelligence in front of you while the migration goes live, and then move quickly to attack any problems that arise.

By the way, if you think you’re immune to site migration problems, then listen to the episode of Google’s Search Off The Record podcast where John Mueller, Gary Illyes, Cherry Prommawin, and Martin Splitt talk about the migration of the webmaster central site to the new search central site. It ends up they ran into several problems just like any other site owner could and had to move quickly to rectify those issues. So, if it can happen to Google, it sure can happen to you. :)

Adding Google’s Crawl Stats Reporting To Your Site Migration Checklist:
There are plenty of checklists and tools out there to help with site migrations. For example, Google’s testing tools in GSC, third-party crawlers like Screaming Frog, DeepCrawl, and Sitebulb, site monitoring tools, log file analysis tools, and more.

And on the topic of log files, they provide the quickest way to understand how Google is crawling your site post-migration. You don’t need to wait for data to populate in a tool, you don’t have to guess how Google is treating urls, redirects, etc., and there are several log file analysis tools hungry to consume your logs.

But there’s a catch… trying to get log files is like attempting to complete a mission as Tom Cruise in one of his great Mission Impossible movies. If gaining log files was a scene in Mission Impossible, I could hear Tom now:

“Wait, so we have to scuba dive under a bridge heavily guarded by troops, climb a 200 story building in our underwear (in 20 degree weather), use elaborate yoga moves to dodge a scattered laser security system, steal the ancient lamp of Mueller which is protected by special forces, hack into a computer system protected by six layers of ciphers, download the log files, and then parachute off the building back into the water, only to scuba dive back under the bridge to safety? No problem… hold my coffee.”

OK, it’s not that bad, but any SEO that has attempted to get log files from a client knows how frustrating that situation can be. They are huge files, seemingly not owned by one group or person in a company, and you can even find some companies not keeping logs for more than a few days (if that). So, it’s no easy feat to get a hold of them.

What’s an SEO to do?

Meet The New Crawl Stats Report in GSC: A (Pretty) Good Proxy For Log Files
In November of 2020 Google launched the new Crawl Stats reporting in GSC. The reporting is outstanding, and it was a huge improvement from the previous version. The new reporting provides a boatload of data based on Google crawling your site. I won’t go through all of the reports and data in this post, but you can check out the documentation to learn more about each of the report sections.

I’m going to cover what Google considers “site moves with url changes”, which covers domain name changes and url migrations. I’ll focus on domain name changes, but you can absolutely use the new Crawl Stats reporting to troubleshoot url migrations as well.

For domain name changes, you can view crawl stats reporting for both the domain you are moving to and the domain you are moving from. So, using the crawl stats reporting can supplement your current migration checks and enable you to see how Google is handling the migration at the source (the old domain).

And for url migrations, you can also surface problems that Google is experiencing post-migration. It’s not as clear as a domain name change, since you can’t isolate the crawl stats reporting by domain, but it can still help you surface issues based on bulk-changing urls.

Note: There is a delay in the Crawl Stats reporting.
The Crawl Stats reporting lags by a few days, so log files are still important if you want to see a real-time view of how Google is handling a site migration. The reporting updates daily, but lags by 3-4 days from what I have seen. Below, you can see the report was last updated on 2/26/21, but today is 3/2/21.

How to identify problems with domain name changes and url migrations using the Crawl Stats reporting in GSC:
As mentioned above, for domain name changes, you can analyze the Crawl Stats reporting for the domain name you are moving from, and the domain name you are moving to. Below, I’ll cover some of the ways you can use the reporting to surface potential issues.

How To Find The New Crawl Stats Report in Google Search Console (GSC):
First, I know there’s some confusion about where the new Crawl Stats report is located. You will not find the report in the left-side navigation in GSC. Instead, you first need to click “Settings”, find the Crawling section of the page which contains top-level crawl stats, and then click “Open Report” to view the full Crawl Stats reporting.

Now that you’ve found the Crawl Stats reporting, here are some of the things you can find when analyzing and troubleshooting a site migration.

404s and Broken Redirects:
The Crawl Stats reporting for the domain you are moving from will list urls that Google is crawling that end up as 404s. All urls during a domain name change should map to their equivalent url on the new domain (via 301 redirects). By analyzing the source domain name that’s part of the migration, you can view urls that Googlebot is coming across that end up as 404s. And that can help you find gaps in your 301 redirection plan.

For example, you can see the reporting for a site that went through a domain name change below. 4% of the crawl requests were ending up as 404s when most of the urls should be redirecting to urls that return a 200 header response code on the new domain.

If you click into that report, you can see a sample of the top 1,000 urls with that issue and you can inspect the urls as well:

And here is what it should look like. 100% of the requests are 301 redirecting to the equivalent urls on the new domain:

Important (and often confusing) note: It’s worth noting that GSC reports on the destination url, so a 404 showing up for the old domain name could actually be showing you a redirect to the new domain name, but that new url 404s. In other words, the 404 is actually on the new domain, but shows up in the reporting for the old domain name. That’s extremely important to understand overall with GSC, and it can cause confusion while analyzing the reporting. I tweeted about this in January with regard to the Coverage reporting:

Image Search: Googlebot Image
If image search is important for your business, you will definitely want to review the “By Googlebot type” section of the reporting. You will see a listing for “Image”. You can click into that reporting to see the urls Googlebot Image is crawling. If you see 404s, 5XX, etc., then make sure you jump on those issues quickly. You should see plenty of 301s if you redirected images properly during the migration (which you should). I covered that in my Mythbusting video with Google’s Martin Splitt about site migrations. The video can be seen later in this post.

As you can see below, Googlebot Image is coming across 404s as well. This is from the site that went through a domain name change.

This is what you should see. Notice how the Googlebot Image requests all properly 301 redirect to the images on the new domain:

Robots.txt issues:
In the host issues section, you can see if Googlebot is having problems accessing the robots.txt file for the domain(s) involved. If Google cannot fetch the robots.txt file (which returns a 200 or 403/404/410), then it will not crawl the site at that time. Google will check back later to see if it can fetch the robots.txt file. If it can, then crawling will resume. You can read more details about how this is handled on Google’s support page (or in the screenshot below). Note, you can 404 a robots.txt file and that’s absolutely fine. This is about Google having problems fetching the file (i.e. Google seeing a 429, 5XX).

And here is what it can look like in GSC’s Crawl Stats reporting. Although this falls under an “acceptable fail rate”, I would sure check why the robots.txt fetch is failing at all:

Other host issues: DNS Resolution and Server Connectivity:
Along the same lines, you can see if there are other host-level issues going on. The host reporting also contains DNS resolution errors and server connectivity problems. You obviously want to make sure Google can successfully recognize your hostname and that it can connect to your site.

Performance Problems:
The reporting also will show pages that are timing out for some reason, so keep an eye on that report. You will find that in the “By response” section.

Subdomain Issues:
Hopefully you picked up all subdomains that were in use before you pulled the trigger on the domain name change. But if you didn’t, you can see Crawl Stats reporting per subdomain that Google is crawling. The catch is that you need a domain property set up in GSC for the domain you are moving from (unless you had those subdomains verified and set up already in GSC).

If you did, you could view the crawl stats reporting for those subdomains separately. Domain properties make this easier since all subdomains being crawled by Google (the top 20 over the past 90 days) will be shown in the Hosts report in the Crawl Stats reporting.

Below, you can see that the crawl stats reporting shows 17 different subdomains with crawl requests over the past 90 days.

Note, I always recommend having a domain property set up. It’s amazing how many companies have not done this yet… If you haven’t, I would do that today. It doesn’t take long to set up and it covers all protocols and subdomains.

Crawl Stats For Site Migrations: Final tips And Recommendations
The Crawl Stats reporting can help site owners and SEOs get closer to log file analysis, when gaining those logs might be tough. Although there’s a lag in the data populating (3-4 days), the Crawl Stats reporting can sure help surface problems during domain name changes and url migrations. And the quicker you can nip those problems in the bud, the less chance they become bigger issues SEO-wise.

Here are some final tips and recommendations:

  • Set up domain properties for each of the domains involved in the migration (if changing domain names). This will give you access to all subdomains in the Crawl Stats reporting.
  • Once data starts populating in the Crawl Stats reporting post-migration, dig into the domain you are moving from. You might see a number of issues there based on what I explained earlier. For example, 404s, performance issues, robots.txt problems, and more.
  • Nail the redirection plan. If you see gaps and problems with your 301 redirects, move quickly to rectify those problems. Nip those problems in the bud.
  • Check for host-level problems (like robots.txt fetch issues, DNS resolution issues, and server connectivity problems). Your redirection plan doesn’t matter if Google can’t successfully connect to your site.
  • Look for pages that are timing out. This would show up in the “By response” section of the reporting. If you see that, dig into those problems to see why the pages are timing out. Again, move quickly to address performance issues.
  • Don’t forget your images! Make sure to 301 redirect your images and then check the section labeled “By Googlebot type”. Then check the “Image” reporting to see how Googlebot Image is crawling your content.

More About Site Migrations: Mythbusting Video
If you are interested in site migrations, then you should check out the Mythbusting video I shot with Google’s Martin Splitt. In the video, we cover a number of important topics including domain name changes, url migrations, redirecting images, when a site should revert a migration, site merges, the Change of Address Tool in GSC, and more.

Summary – GSC’s Crawl Stats as a proxy for log files.
After reading this post, I hope you see the power in adding Google’s Crawl Stats reporting to your site migration checklist. The reporting provides a boatload of great information based on Google crawling your site post-migration. I’ve found it extremely helpful while helping companies monitor and troubleshoot domain name changes, url migrations, and more. And remember, Murphy’s Law is real for site migrations. Things will go wrong… which is ok. The important part is how quickly you handle and rectify those problems.

GG

Google Search Console (GSC) reporting for Soft 404s is now more accurate. But where did those Soft 404s go?

By Leave a Comment

On January 11, 2021 Google announced improvements to GSC’s Index Coverage data. With the update, which is clearly apparent in the reporting, Search Console removed the generic “crawl anomaly” issue (and mapped those to more granular issues), changed how urls are reported that are blocked by robots.txt, added a new issue called “indexed without content” as a warning, and then had one final note about soft 404s. That last bullet said, “Soft 404 reporting is now more accurate.” And based on my work with large-scale sites, I was particularly interested in that final bullet.

A number of the sites I help had tens of thousands, or even hundreds of thousands of soft 404s being reported based on the nature of their content. Starting on January 5, 2021, the number of soft 404s reported in GSC fell off a cliff. I’m talking about a severe drop, like these examples:

In order to understand why alarms were going off in my office, and why I’m particularly interested in those drops, let me quickly cover more about soft 404s. Then I’ll explain what I found once I dug into each situation.

A Deeper Look at Soft 404s:
Google explains that a soft 404 is a url that tells the user the page doesn’t exist, but it still returns a 200 header response code (instead of a hard 404). In practice, the soft 404s that Google has surfaced historically have been urls that span different page types. For example, the most obvious is a product that’s no longer available that still returns a 200 response code. It might have very little content and a message telling users the product is not available on the site anymore.

But, other types of pages have showed up in the soft 404s reporting over time. For example, I’ve seen super-thin pages show up there like articles, slideshows, job listings, pages with images or video with little textual content, and more. And on larger-scale sites, those pages can be problematic quality-wise depending on how Google is handling them.

For example, a soft 404 is treated as a hard 404. Google will not index those urls identified as soft 404s. But, if a site had a big soft 404 problem, I often found that Google wasn’t always handling all of those pages types as soft 404s… And that led to many thinner urls getting indexed.

So, if you had 50K super-thin pages and Google saw them as soft 404s, then technically that shouldn’t be a huge problem quality-wise. Since they are seen as hard 404s, Google will obviously not take them into account when evaluating quality.

But, what if suddenly those urls aren’t seen as soft 404s anymore? Well, those urls could be counted when Google evaluates quality overall for the site. That’s a big reason my “quality antennae” went up when I saw tens of thousands, or even hundreds of thousands, of soft 404s suddenly disappear. Are they indexed now, categorized as another issue in GSC and not being indexed, or maybe not even known to Google anymore? I NEEDED TO KNOW!

In addition, soft 404s can impact crawl budget, which can be a concern for larger-scale sites. Note, most sites do not need to worry about crawl budget. Only very large sites (typically sites with over one million urls) need to worry about crawl budget. For example, this site needs to worry about crawl budget:

Google’s own documentation explains that soft 404s can eat up crawl budget. So it’s another concern when it comes to soft 404s (and understanding where they magically went on 1/5/21). In the past, if you saw them flagged in GSC, then you could identify patterns and handle appropriately. But if they suddenly disappear, it could be harder to surface and handle.

In Search Of: Lost Soft 404s
This all leads me back to one question… where did all of those soft 404s go?? After seeing the number of soft 404s drop off a cliff on 1/5/21, I decided to dig into the data across sites and find out what happened. So, I grabbed my virtual pickaxe and headed into the GSC mine. I was determined to find out how those urls were being treated now.

Now, if you choose to dig into search console after a change is rolled out by the GSC product team, you’ll quickly realize that you cannot access historical data. That sucks and means you will have a hard time finding urls previously categorized as soft 404s unless you documented them somehow. I had this covered since I always dig heavily into GSC data when auditing client sites. I was able to open my vault of recent audit documents and resurface many soft 404s that were reported over the past six to twelve months.

Based on digging into the data, I was sometimes relieved with what I discovered for certain situations, but also alarmed by what I found in other situations. I’ll explain more below.

Where Did Soft 404s Go and How Is Google Currently Handling Those URLs?
Below, I’ll cover the various categories of findings based on digging into a number of large-scale sites that previously reported many soft 404s. On 1/5/21, the number of soft 404s dropped off a cliff and I was determined to find out how they were currently being treated.

Super-interesting: “The URL is unknown to Google”
This was incredibly interesting to me. I noticed many url across sites that were once reported as soft 404s now returning “The URL is unknown to Google” when inspected in GSC. It’s almost like Google completely wiped out the history for these urls.

This isn’t a category that shows up in the Coverage reporting for obvious reasons… since it’s NOT known to Google. But, it was known to Google at some point recently. I know that for sure. Google’s documentation does explain more about this categorization, but I’m not sure it answers why the soft 404s are now “unknown to Google”. Here is what Google’s documentation explains:

“If the index coverage status “URL is unknown to Google” appears, it means that Google hasn’t indexed the URL either because it hasn’t seen the URL before, or because it has found it as a properly marked alternate page, but it can’t be crawled.”

I’m going to reach out to Google to learn more about this. I would imagine that Google handling many urls as this category is helping it improve the efficiency of its own systems, but again, I’ll see if Google can comment about this. But if it’s unknown to Google, it won’t hurt your site quality-wise. That’s good.

“Crawled, not indexed” and “Discovered, not indexed”:
I have covered this category a number of times in my other posts about technical SEO and broad core updates. Both “Crawled, not indexed” and “Discovered, not indexed” can signal quality problems and/or crawl budget issues. With “Crawled, not indexed”, Google discovered and crawled the url, but has chosen to not index it. I have often found low-quality or thin content when analyzing this category across sites (or just parts of a site Google isn’t digging for some reason).

So, if your soft 404s end up as “Crawled, not indexed”, then that’s ok in the short-term (since they aren’t being indexed and can’t be held against the site quality-wise), but I would still heavily dig into the urls and handle them appropriately. This is also one of the reasons I believe we need larger exports from GSC’s Coverage reporting! Exporting this category in bulk would help site owners identify more patterns across the site that could be causing problems.

And for “Discovered, not indexed”, Google knows about the url, but has decided to not even crawl it (and clearly not index it). I have seen quality problems and problematic url patterns while analyzing this category of urls. For example, with some large and complex sites with deep quality issues, it’s like Google knows ahead of time what it’s going to find on many of those urls (based on url structure) and doesn’t even want to crawl them.

So, if you see many soft 404s now being categorized as “Discovered, not indexed”, then that’s also ok in the short-term since they can’t be held against the site quality-wise if they aren’t being indexed. But just like with “Crawled, not indexed”, I would dig in and make sure the urls are being handled properly. If they should 404, then make sure they return hard 404s. If they are thin and low-quality, then boost the content there. And if it’s a problematic area of the site, then work on improving that area overall. Remember, “Discovered, not indexed” means Google knows about the urls, but doesn’t believe it’s worth it to even crawl them (let alone index them).

URLs Now Indexed: Danger Will Robinson! Danger!
Yes, this is a super important category. When digging into urls that were previously categorized as soft 404s, I noticed some of them were now being indexed! That can be a dangerous situation for some sites. For example, if they were thin and low-quality urls that Google is now indexing for some reason, those urls will now be taken into account while Google evaluates quality for the site overall. “Quality indexing” is very important (especially for larger-scale sites), so having an influx of low-quality content get indexed is not a good thing.

If you notice many urls that were once flagged as soft 404s now being indexed, then it’s important to handle those urls appropriately now. For example, you can noindex low-quality content, boost the content there (improve it so it meets or exceeds user expectations), 301 it to a newer version of the content, or 404 the pages if they shouldn’t be on the site anymore. But I would not leave them as-is if they are low-quality or thin.

True soft 404s still seen as soft 404s:
Even though many soft 404s disappeared from the reporting in GSC, there were still urls categorized as true soft 404s. For example, pages that were literally blank that returned 200 header response codes, pages that returned “not found” messages on the page, but returned 200 response codes, etc. Again, true soft 404s. I guess this is the “handling soft 404s more accurately” part from Google. For these urls, I would make sure to handle them appropriately. If they should 404, then have them return hard 404s and not 200 header response codes. And if they shouldn’t be seen as soft 404s, then fix or boost the content there.

Moving Forward: What Site Owners Can Do
If you have seen a big drop in soft 404s starting in early January 2021, then I highly recommend digging into the situation. If you don’t have historical data containing urls that were once categorized as soft 404s, there’s not really much you can do to see how those soft 404s are being reported now. But, if you did document soft 404s for your site, then you can inspect those urls to see how Google is handling them now.

Are they “Unknown to Google” now, are they being categorized as “Crawled, not indexed”, “Discovered, not indexed”, are they still soft 404s, or are they being indexed? And if they are being indexed, then they can be counted when Google evaluates quality for the site overall. That’s why it’s important to handle the urls appropriately.

Summary – Know Your Soft 404s
With the latest data improvements in GSC, Google started better handling soft 404s. That’s great, but many sites saw a big drop in urls reported as soft 404s at that time. Since that category in GSC often yielded thin or low-quality content on a site, it’s important to understand how Google is currently handling the urls. Are they still not being indexed, or are they now being indexed and counting towards quality? I recommend digging into the situation for your site and then handling those urls appropriately.

GG

Google’s December 2020 Broad Core Algorithm Update Part 2: Three Case Studies That Underscore The Complexity and Nuance of Broad Core Updates

By Leave a Comment

In part one of my series on the December 2020 broad core update, I covered a number of important items. If you haven’t read that post yet, I recommend doing that and then coming back to this post. For example, I covered the rollout, timing, tremors and reversals, the impact I was seeing across verticals, and then I covered a number of important points for site owners to understand about broad core updates.

Those items spanned quality indexing, the importance of providing a strong user experience, avoiding hammering users with ads, technical SEO problems that cause quality problems, the A in E-A-T, machine learning in Search, and more. My first post will provide a strong foundation before jumping into the case studies I’ll cover below.

Three Case Studies, Three Interesting Lessons About Broad Core Updates
In this post, I’ll cover three case studies that underscore the complexity and nuance of broad core updates. Each is a unique situation and site owners (and SEOs) can learn a lot from what transpired with the sites I’ll be covering. I was planning on covering four cases, but this post was getting too long. I might cover that other case in another post in the future, though.

Remember, Google is evaluating many factors over time with broad core updates. It’s never about one or two things. Instead, there could be many things working together that cause significant impact. Google’s Gary Illyes once explained that Google’s core ranking algorithm is comprised of “millions of baby algorithms working together to output a score”. That’s why I recommend using a “kitchen sink” approach to remediation where you surface all potential problems impacting a site and fix them all (or as many as you can). The core point, pun intended, is to significantly improve the site overall.

It’s also important to understand that Google wants to see significant improvement over the long-term. Google’s John Mueller has explained this many times over the years, and it’s what I have seen while helping many companies deal with broad core updates. And the case studies below support that point as well.

In the case studies below, I’ll cover each site’s background leading up to the December broad core update, some of the important issues I surfaced while helping those companies, how those issues were addressed (or not), and what the outcome was when the December update rolled out. Again, my hope is that these cases will help site owners understand more about broad core updates, how to deal with negative impact, and understand various factors that could be leading to negative impact. Let’s jump in.

Below, I have provided a table of contents in case you want to jump to a specific part of this post:

Case Study 1: Niche News Publisher – A Double Core Update Hit, But E-A-T Galore
The first case I’m going to cover is a super interesting one (and I felt especially connected to it since I loved the site even before the site owner reached out to me!) It’s a news publisher that focuses on a very specific niche that reached out to me after getting hit hard by the January 2020 core update. Again, I knew the site right away (since I had been reading it for years) and I was initially extremely surprised to hear the site had been hit hard.

Note, they contacted me in March of 2020, so months after the January update rolled out. Waiting to improve a site is a big risk site owners take and I’ll touch on that later in this case.

The site had lost 45% of its Google organic traffic overnight with the January 2020 broad core update:

I wish that was the only drop the site had to deal with in 2020, but it wasn’t. More about that soon.

Google News, Top Stories, and Discover:
Based on how Google was viewing the site, it rarely showed up in Top Stories, almost never ranked highly in Google News, and had no Discover traffic at all. And I mean none… the Discover reporting didn’t even show up in GSC. That means it had no impressions or clicks over time. And remember, it’s arguably the top publisher in its niche.

E-A-T Galore:
The site, including its core writers, had E-A-T galore. That was definitely not the problem, not even close. It’s a well-known, well-respected, leader in its niche. I would argue it’s the top news site in its niche and has broken some of the biggest stories in its category.

With regard to the A in E-A-T, which is authority, Google has explained that one of the best-known signals it uses is PageRank, or links from across the web. Google explained this in a whitepaper it released about fighting disinformation.

Also, Gary Illyes once explained that E-A-T was largely based on links and mentions from well-known sites. Note, I covered the A in E-A-T in my first post in this series (if you are interested in learning more about that topic). For this site, this was covered big-time. The site is absolutely an authority in its niche. The site has over 2M inbound links from over 26K referring sites. And a number of the sites linking to this news publisher are some of the most powerful and authoritative websites in the world.

So what was the problem? Why did the site take such a big hit? If you read part one of this series on the December 2020 broad core update, then you know that core updates are never about one thing. Google is evaluating many factors across a site and for an extended period of time. I always say there’s never one smoking gun, there’s typically a battery of them.

Therefore, I dug in the way I normally would in order to surface all potential issues that could be causing problems. That included auditing content quality, user experience, the advertising situation, technical SEO, and more. I clearly explained this process to my client and they were on board. They said they would clear a path, implement as many changes as possible, and as quickly as possible, in order to see recovery in the future.

Well, Hello May 2020 Broad Core Update, How Nice To Meet You…
It’s also worth noting that as I was auditing the site, and my client was implementing changes, the May 2020 core update rolled out AND THEY GOT HIT AGAIN. This can absolutely happen if a site hasn’t improved significantly (or if a site chooses to not improve at all). In addition, even if you are implementing changes, it’s important to understand that recent changes would not be reflected in the next broad core update. I covered that several times in my previous posts, including both the May 2020 core update and part one about the December 2020 broad core update.

With the May 2020 core update, this news publisher dropped by an additional 51%. That’s on top of dropping 45% with the January core update. So they were down over 90% from where they once were:

Needless to say, my client felt defeated. They couldn’t believe they dropped even more. They were now receiving a trickle of traffic compared to what they used to. I explained that they should continue to make changes, that not enough had been implemented, and not over the long-term.

I basically felt like Mickey in Rocky pushing his fighter to keep driving forward:

Needless to say, my client was eager to keep driving forward, implement big changes, improve the site significantly, etc. But, it’s important to know that you can get hit by multiple core updates in a row if Google isn’t digging your site. I have covered that before in blog posts and in my presentations on broad core updates.

Below, I’ll cover several of the core items I surfaced during the audit that could have been contributing to the drops the site experienced during two broad core updates. I can’t cover everything that was addressed during the engagement, but I will provide a number of important findings. And keep in mind, there are millions of baby algorithms working together to output a score. It’s not about one of the following items, but more about improving all of them (including other items not covered below).

Lower-hanging fruit: Thin content, cruft:
With many news publishers, there’s a tendency for low-quality content to build up over time. They are typically larger, more complex sites and cruft can definitely build. And after years go by, some sites might have a lot of lower-quality content in pockets across the site.

Although this site produces some of the best content in its niche, I did surface a lot of cruft that had built up over time. I explained what “quality indexing” was to my client, which is making sure only high-quality content is indexed, while ensuring low-quality or thin content is not indexed. They understood the concept quickly and had no problem nuking thin and low-quality content from the site. Remember, Google is on record explaining that every page indexed is taken into account when evaluating quality.

Here is Google’s John Mueller explaining this:

Content quality (and strategy):
As I was analyzing content that dropped heavily during each update, I noticed something interesting. There were a number of round-up-like posts that were thin and somewhat misleading. So instead of creating one strong page that could be updated for the content at hand (which I can’t reveal), the site was publishing many thinner pages over time. This led to many pages on the site that were ranking and receiving traffic, but had no way to meet or exceed user expectations. This was a huge find in my opinion.

My client tackled this head-on and used the approach I recommended for creating one piece of content that can live over time and always contain the latest information, while archiving older pages when needed. This led to higher quality content that can meet or exceed user expectations, while also improving the “quality indexation” levels on the site overall. It was a win-win.

User Frustration and “Hell hath no fury like a user scorned”:
While continuing to analyze content that dropped on the site, I noticed some content that could easily confuse users (and it looked like it was confusing Google too). I can’t go too in-depth here, but let’s just say that users searching Google to accomplish a task were ending up on the site and on these pages. But the pages did not provide that functionality at all.

So users were visiting that page, then digging further into the site believing they could ultimately find what they were looking for. But they couldn’t find that information no matter where they turned, which led them back to the original landing page. And then using behavior flow in Google Analytics, I could see they were then trying other pages on the site in a never-ending journey. This is similar to the case study I wrote about the engagement trap and user frustration.

After explaining this to my client, and showing them behavior flow screenshots of what was happening, they nuked those pages. I also believe this was an important finding. Remember, “hell hath no fury like a user scorned”.

UX, Unavailable Videos, and Broken pages:
There are also times where news publishers push an article live and don’t revisit those pages (since they are working on the next big story). Unfortunately, those older pages can break for one reason or another. For example, I surfaced pages with either missing images or videos that weren’t available anymore. And when the videos were core to the article, that made the pages useless. There is no way a page can meet or exceed user expectations when the core element doesn’t even load.

So in an effort to enhance the user experience across the site, my client tackled these issues page by page. And if a page couldn’t meet or exceed user expectations anymore based on query, then the page was nuked.

Here was a third-party video used to support an article, but it was no longer available (and was actually removed due to a copyright claim against whoever uploaded the video to YouTube):

Sponsored links: Handle with care:
I’ll keep this one quick. The site has sponsors that are prominently displayed on the site with sponsored links. But, it wasn’t clear they were sponsors, so that required a clear disclaimer. Second, the links were followed. My client nofollowed all of those links and used rel sponsored as well. This was an easy fix, but the right one on several levels.

Google canonicalizing pages to third-party sites:
There was also a canonicalization issue where the publisher had uploaded a number of pdfs from other sites to support content from specific articles. In Google Search Console (GSC), you could clearly see that Google was canonicalizing all of those pdfs to pages on other sites (knowing where the original pdfs originated).

Having a few here and there isn’t a big deal, but you have to wonder what type of signal it sends to Google when many are seen as duplicate (and Google is choosing pages from third-party sites as the canonical urls). My client determined the pdfs were not essential to the articles and nuked many of them from the site.

In addition, there were a number of pages that had canonical tags pointing at urls that didn’t resolve with 200 codes. Some were 404ing, others were redirecting, etc. Rel canonical is a strong signal to Google about which pages should be indexed and ranking, so I always recommend having accurate canonical tags.

Here is an example of Google canonicalizing pdfs to urls on other sites:

Downstream Sites and Maintaining User Happiness (and Security):
For larger-scale publishers, it’s easy to not revisit older articles over time, including checking the outbound links on those pages. And over time, you could be driving users to very strange (and risky) sites. I’ve always recommended auditing outbound links to make sure you aren’t putting your users, and essentially Google’s users, in a tough situation.

Based on the niche, this news publisher had a number of links that led to sketchy websites, and I would say even risky websites. It’s hard to say if those domains changed hands at some point, or if something else was going on, but the outbound link situation wasn’t great. My client worked on making sure outbound links were accurate, helpful, and didn’t put users at risk.

Chrome was even showing a safe browsing warning when visiting some of the downstream sites:

Search URLs and playing hide and seek with robots.txt:
When I began auditing the site, I quickly noticed that many internal search urls were showing up in the crawl data, in GSC, etc. Some were indexed, some were not, and the site’s robots.txt was supposed to be blocking them (or I thought anyway). It ends up there were many internal search urls triggered from outside the site that didn’t return any content. I saw thousands of these urls in the reporting.

But… the directory holding internal search urls was blocked by robots.txt. So what was going on?? Well, I wrote a post last year that covered how robots.txt files are by protocol and subdomain, and that’s exactly what was going on here. Basically, you can have multiple robots.txt files running at one time and the directives in each file can conflict. For this site, there were two files running on the same subdomain (based on protocol), and one blocked internal search and the other didn’t. My client cleaned this up pretty quickly.

Beyond what I covered above, there were a number of other findings surfaced during the audit. I can’t cover everything here or the post would be huge. But just understand that they tackled many different problems across the site. Some major, some minor, but all with the goal in mind of improving the site significantly over the long-term.

Liftoff in December: 149% Surge
Months went by since the last broad core update in May and my client would email me periodically to see if I had any idea when the next update was coming. I explained there’s no set timeframe for broad core updates and that we would hopefully see one soon. I told them to keep driving forward.

And then nearly seven months after the May update, the December 2020 broad core update rolled out. And it was a very, very good one for this news publisher. They surged almost immediately, and it was hockey stick growth. The site surged 149% with the update:

But that doesn’t even do the surge justice. Remember, they had almost no rankings and traffic from Google News traffic, Top Stories, or Discover. After the update, they started ranking in Top Stories and in Google News more prominently. It was awesome to see. Here is the news tab in GSC showing the growth there:

Regarding Discover, that is still a mystery to me. They still have no Discover traffic at all. I believe it could be due to the niche they cover and not necessarily the site, but it’s hard to say for sure. So, this would be the one area that didn’t surge with the update.

Case Study 2: Crushed By The December Update: It’s Like Google Doesn’t Even Know You Anymore
The next case study I’ll cover is a site that has seen its fair share of movement over the years (dating back to medieval Panda). It has been doing well recently, and even saw a nice surge during the May 2020 core update. But there was a growing, even insidious, problem that could have heavily contributed to the drop the site experienced with the December broad core update.

I can’t go into too much detail about the niche, but it’s an affiliate site with several types of content (which target a different user intent). The December update was not a good one for this site… It lost 57% of its Google organic traffic overnight starting on 12/3/20 and Google represents a majority of overall traffic. Needless to say, this is extremely problematic for the business.

I know the CEO of the company and he reached out quickly after the December update started rolling out. The drop, as you can see above, was dramatic. I had a call with the CEO, and as I learned more about their situation and setup, my “broad core update antenna” went up. I knew the site pretty well, and specific points he was making about what they had been doing recently had me concerned. I’ll cover what I can below. My hope is that they will be making big changes moving forward.

Swerving Out Of Your Lane: Major injection of thin content (and content not targeting their core competency):
I’ve mentioned the importance of “staying in your lane” several times over the years after seeing companies expand beyond their core competency and publishing content that they either didn’t have the expertise to write about, or content that could confuse Google topicality-wise. For this site, the latter was my concern.

And in addition to potentially confusing Google’s algorithms about what the site’s core competency was, a lot of the new content was thin and lower quality. This is exactly the type of content that sites should avoid having indexed. I’ve covered “quality indexing” many times in the past and this is exactly the type of content I would recommend keeping out of Google’s index. Unfortunately for this site, thousands of pages were added over time and Google was actually ranking some of it well. That’s a dangerous situation and could end up very badly SEO-wise.

Before they knew it, a big shift had happened percentage-wise for the site from an indexing standpoint. Now there was more low-quality and thin content on the site than core content. And that core content is what got them to where they were SEO-wise (performing well). And in my opinion, Google is clearly not digging that shift. Again, the site dropped by 57% since the December broad core update rolled out.

To add insult to injury, many of those thin pages contained content that expired, yet the pages remained on the site and indexable. Google was seeing some of those pages as soft 404s, while others remained in the index. So, if users were searching Google and finding those pages, the content had no chance of meeting or exceeding user expectations. Again, this was an insidious problem in my opinion. And those thin pages were causing multiple problems from an SEO standpoint.

Here was a quick site query showing close to 1,500 urls indexed that held expired content:

Beyond those findings, there were also a number of other issues that I surfaced which should be fixed. Those included the use of popups on some versions of the pages (including on mobile), more thin content across other page types, urls categorized as “crawled, not indexed” that should be handled properly (which spanned several page types), and more.

I’m still in the middle of analyzing the site, so there will be more findings. But the core point with this case is to always maintain strong quality indexing levels, make sure you don’t tip the scales with lower quality content, always make sure you can meet or exceed user expectations, and make sure technical SEO problems don’t end up causing quality problems. I hope to report back about this site after subsequent broad core updates (based on the changes the site owner decides to implement).

Case Study 3: Overcoming “The Creep”
A large-scale site in a tough niche roars back with the December update:
I can’t tell you how happy I am to be covering this recovery. This was actually one of the case studies in my post about the May 2020 core update, but as a big drop! As I mentioned in that post, many problems had crept back into the site over time. They had surged with previous major algorithm updates only to get hammered by the May 2020 core update. They dropped 41% overnight in early May.

Revisiting Core Problems: Bombarding Users With Ads
Leading up to the May broad core update, the ad situation had gotten much, much worse over time. It was extremely aggressive, disruptive, and even deceptive in some cases. This was not a new problem for the site since they had tackled aggressive ads previously (years ago). But like I said earlier, some bad problems crept back in over time.

For some pages, there were 15-20 ads per page and organized in a way that was extremely disruptive to the user experience. After sending my findings through (including many screenshots of what I was seeing), the site owner moved quickly to tone down the ad situation. That resulted in a reduction of 50% of ads per page in some situations.

Needless to say, this improved the user experience greatly on the site. And those aggressive ads were impacting performance as you can see in the Core Web Vitals reporting in GSC. CLS scores were not good. The number of urls with CLS problems is now down to 0 post-update (after the site owners worked on toning down the ad situation and improving performance overall). They also moved to AWS after the May update to improve performance for the site overall.

Large and Complex Site With Pockets of Thin Content:
In addition, there was also a thin content problem (which is common in this niche). This is also something that has been brought up in the past for this site. Once the pockets of thin content were uncovered, the site owner moved quickly to noindex or 404 as much of that content as they could (depending on the page type). This led to an increase in “quality indexing”, which is making sure only high-quality pages are indexed, while ensuring low-quality or thin content is not indexed. This is something I have written about many times and it can rear its ugly head easily on larger-scale and complex sites.

Here are some screenshots of thin content I was surfacing. Also, notice the ads + thin content in the second screenshot. That’s a lethal combination in my opinion:

The Power of Providing Unique Content For Sites With The Same, Or Very Similar Content:
In my post about the December 2020 core update, I mentioned how sites with the same, or very similar content, need to differentiate their sites as much as possible. John Mueller has covered this several times in webmaster hangout videos and I have helped a number of companies deal with this situation.

For this site, I recommended adding some type of unique and valuable content beyond what every other site was providing. Once I ran through the reasons why they should consider this, along with sending John’s video covering the issue, the site owner brainstormed a few ideas and came up with a strong solution. I can’t explain what they did specifically, but they were able to add unique content in a prominent place on core pages that was helpful for users.

Percentage-wise, this didn’t impact a large percentage of pages on the site, but it did impact some of the most popular pages. I’m hoping they roll this out to more pages as time goes on.

Here is Google’s John Mueller explaining the importance of differentiating your content as much as you can:

Technical Problems Causing Quality Problems: Mobile and Desktop Parity Issues
When digging into the site last May, I started noticing a problem with the mobile setup. This site still runs an m-dot subdomain holding the mobile pages, and I started checking for parity across mobile and desktop. What I found was alarming across some urls.

When switching to mobile, the urls were redirecting to the desktop version, but to a different url! This wasn’t site-wide and was just impacting certain urls, but needless to say, this was extremely problematic. Also, Google still hasn’t moved the site to mobile-first indexing based on the m-dot setup and the parity problems I discussed in my May 2020 case study post. For example, the pages lacked parity from a content standpoint, links standpoint, and more. So, Google’s systems are clearly having issues with moving the site to mobile-first indexing. And to clarify, there’s no ranking benefit to being moved to mobile-first indexing, but not being moved underscores the parity problems I just covered.

Since the desktop content is what’s indexed for this site, it’s hard to say how the weird redirect problem was impacting the site SEO-wise, but it’s clear this was impacting users. Once I communicated the problem to the site owner, they tackled the strange redirect problem quickly. But who knows how long that problem was in place, how many users that impacted, etc.

Beyond what I covered above, there were a number of other problems riddling the site (across several categories of issues). There were more quality problems, UX issues, technical problems causing quality problems, and more. Again, it’s a large-scale and complex site with many moving parts (and it’s in a tough niche). The site owner moved to make as many changes as possible.

And by the way, they still have a huge change in the works. As I mentioned earlier, they run an m-dot now (which is not optimal), but they have a responsive version of the site ready to launch soon! So theoretically, things will get even better over time usability-wise, site maintenance-wise, etc. Needless to say, I’m eager to see that go live.

Happy Holidays! The December Update Comes Roaring Through…
Since you can’t (typically) see a recovery in between broad core updates, the site owner fixed as much as they could and kept driving forward. The site owner was fully aware that they probably needed to wait for the next broad core update to see significant movement in the right direction and hoped for the best.

Well, the December update didn’t disappoint!

Within 24 hours of the rollout, the site absolutely started to surge. The site is up 40% since the December update. It’s been amazing to watch and it underscores how a “kitchen sink” approach to remediation worked well for this client.

Also, they are moving to a responsive design soon, which will alleviate parity problems across mobile and desktop and hopefully get them moved to mobile-first indexing soon. Again, there’s no ranking benefit to getting moved to mobile-first indexing, but it decreases the number of issues that can arise based on maintaining two pages for every one on the site. Also, mobile-first indexing will be enabled for all sites in March of 2021, so I’d love to see them with a solid setup and possibly moved before that date.

Moving Forward: Another List of Tips and Recommendations for Site Owners
As you can see via the case studies I covered, broad core updates are complex and nuanced. Google is evaluating many factors across a site and over the long-term. When a site is negatively impacted by a broad core update, there’s never going to be one thing a site can surface that’s causing the drop. Instead, it’s often a combination of issues that is dragging the site down.

Below, I’ll cover some final bullets for site owners that are dealing with negative impact from broad core updates.

  • Relevancy adjustment? – First understand whether or not there was a major relevancy adjustment, which could be correct. For example, was your site ranking for queries it had no right ranking for? There’s not much you can do about that… But if it’s not a relevancy adjustment, then dig into your site to surface all potential issues that could be causing problems.
  • The power of user studies – To understand how real users feel about your site, content, expertise, etc., run a user study through the lens of broad core updates. I’ve written a case study about doing this that you can review, Google recommends running user studies for this purpose, and it makes sense on multiple levels. I can’t tell you how powerful it can be to receive objective feedback from real users.
  • Never one smoking gun – Avoid looking for one smoking gun. You won’t find it. Instead, objectively analyze your site and surface all potential problems that could be causing issues. Then form a plan of attack for fixing them all (or as many as you can).
  • Recovery – Understand you (typically) cannot recover until another broad core update rolls out. I have covered this in previous posts and in my SMX Virtual presentation about broad core updates. So, don’t roll out the right changes only to roll them back after a few weeks. That’s not how this works. You will end up spinning your wheels.
  • Read the Quality Rater Guidelines (QRG) – There’s a boatload of amazing information directly from Google about what they deem high versus low quality. The document is 175 pages of SEO gold.
  • Recovery Without Remediation – Understand that some sites recover without performing any remediation. That can definitely happen since there are millions of baby algorithms working together to output a score. In my opinion, if you don’t significantly improve your site over the long-term, you leave your site susceptible to seeing negative impact down the line. That’s why you can see the yo-yo effect that some sites experience (as they sit in the gray area of Google’s algorithms). That’s a maddening place to live for site owners. I would try and improve as much as possible.

Summary – Google’s Broad Core Updates Are Complex and Nuanced
I hope you found the case studies I covered interesting. One thing is for sure, they underscore the complexity and nuance of Google’s broad core updates. If you have been negatively impacted by a broad core update, then I recommend reading several of my posts about core updates, form a plan of attack for thoroughly analyzing your site through the lens of those updates, and then work to significantly improve your site over time. You can absolutely recover, but it takes time, hard work, and patience. Good luck.

GG

Back To Table of Contents

Read part one of this series on the December 2020 broad core update.