While helping companies that have been heavily impacted by Google’s broad core updates, the topic of “authority” comes up often. And that’s especially the case for companies that focus on “Your Money or Your Life” (YMYL) topics. For example, sites that focus on health, medical, financial, etc. all fall under YMYL.

If you’ve read my posts about broad core updates, then you know that Google is taking many factors into account when evaluating sites (and over a long period of time). It’s one of the reasons I recommend taking the “kitchen sink” approach to remediation where you surface all potential issues impacting a site quality-wise, and work to fix them all (or at least as many as you can). For example, improving content quality, user experience, technical SEO problems causing quality problems, the site’s advertising setup, affiliate setup, and more.

And for sites that cover YMYL topics, it’s important to understand that they are held to a higher standard. Google wants to surface the most authoritative sites for sensitive queries (for topics that can impact the health, happiness, financial status, etc. of a user). And that’s one area where authority comes into play.

“Authority” can be a nebulous topic, so I’ll just focus on one aspect for this post – which are links. Danny Sullivan (pre-Google) actually wrote a good post about the subject where he interviewed Paul Haahr, distinguished engineer at Google. In that post, Danny explained how a mixture of factors are used to measure “authority”, with PageRank being the “original authority metric” (links from across the web). So links are not the only factor contributing to authority, but they are important. Instead, there’s a bucket of signals Google is using. By the way, Paul is one of the smartest guys at Google from a ranking perspective. I’ve always said that when Paul talks, SEOs should listen. :)

In addition, Google has explained that PageRank, or links from across the web, is one of the most well-known factors when evaluating authority. That’s from a whitepaper Google published about fighting disinformation across the web.

Google also explained in the whitepaper that when it detects a YMYL query, then it gives more weight in its ranking system to factors like authoritativeness, expertise, and trust. So those links and mentions are even more important for YMYL queries/topics.

Going even further, Google’s Gary Illyes explained at Pubcon in 2017 that E-A-T is largely based on links and mentions from authoritative sites. Here is a tweet from Marie Haynes after Gary explained this at Pubcon:

But there’s more…. In 2020 Bill Slawski covered a super-interesting patent that discusses expertise and how Google might be categorizing sites to determine which set of sites should even be considered for ranking (for certain types of queries). By the way, that sure sounded a lot like what happened with the Medic Update in August of 2018 (when many health and medical sites saw extreme volatility). As part of that patent, E-A-T was mentioned with a focus on “authority” (which we know includes links and mentions from authoritative sites).

As you can see, links are important and can provide a strong signal to Google about the authoritativeness of a page and website. And as Gary Illyes once explained, that includes mentions on well-known sites (and not just links).

A Link Authority Gap Analysis: Comparing The Most Powerful Links Between Sites
Based on what I explained above, it can be important to understand where powerful links and mentions are coming from (both to your site and to your competitors). And that’s especially the case if you focus on YMYL topics. Therefore, I find it’s always interesting to compare link profiles, understand the differences in links from authoritative sites, and identify gaps in your own link profile in order to have a solid understanding of your current state.

The tough part for site owners is understanding that there’s no quick fix to a situation where your competitors have a clearly stronger link profile from an authority standpoint. You can’t just go out the next day and gain links from CNN, the CDC, The New York Times, and other authoritative sites. Note, as I explained earlier, rankings (and “authority) are not just about links, so this isn’t a binary rank-or-not-rank situation. But again, the right links can be a strong signal to Google about the authoritativeness of a piece of content, or a site overall.

For example, it’s not easy to earn links from the following domains…

Also, from a link earning perspective, you typically need to build a strong content strategy, social strategy, and possibly digital PR strategy to help earn links naturally over time. And that takes time… it’s not something you can do quickly (if done naturally). And if you find a large gap from a link authority standpoint, it’s definitely not a reason to spam every news publisher in the world to try and gain links. That typically won’t work well and could backfire big-time.

Peeling back a link profile to reveal “link authority gaps”:
In this post, I’m going to explain how to quickly surface the most authoritative links pointing to a competitor when those same sites don’t link to your own site. In other words, finding the authority gap between sites. The post isn’t about doing a deep dive into a link profile, which can also be beneficial. Instead, it’s focused on quickly identifying important gaps from an authority standpoint.

Again, this is a nuanced topic and just because one site has links from X domain doesn’t mean that’s why it is ranking above you. But, the link authority gap can give you a pretty good view of why a site may be seen as more authoritative than another. In addition, you can learn a lot about the type of content earning those links and possibly identify content gaps on your own site.

Below, I’ll cover how to run a quick link comparison using three tools:

• Majestic Solo Links
• Semrush Backlink Gap
• ahrefs Link Intersect tool

Solo Links in Majestic:
I love Solo Links and it has definitely flown under the radar for a really long time. It compares the top 500 referring domains for two different websites (ordered by Majestic’s Trust Flow metric). I have found it to be a strong way to surface the most powerful domains linking to one site, and not another. And no, Google doesn’t use Trust Flow or any third-party metrics from the various tools. It’s just Majestic’s internal metric for estimating quality based on being closely linked to “trusted seed sites”.

Just enter the two domains you want to compare and click the magnifying glass. You’ll see a Venn digarm representing the overlap, and the gaps, between both sites. Below the Venn diagram, you’ll the referring domains listed, the number of links, and you have the ability to quickly view the links. For example, if you click the link count, you will see the actual links along with supporting information (like anchor text, if it’s nofollow, when it was first seen, etc.) It’s not perfect (no tool is), but it can sure get you moving in the right direction when trying to determine the link authority gap.

For example, here’s a gap analysis for a powerful site in the health niche compared to a smaller, less authoritative site. You can clearly see the differences in the Venn diagram and then in the referring domains list.

A quick note to Majestic:
Please add the ability to export all of the links in bulk and not just the domains. It would be great to simply click “export” and get all of that data in one shot. It’s a bit cumbersome now to get that data out of Solo Links since you can only export links for each domain and not all of the domains listed at one time…

Backlink Gap in Semrush:
Similar to Solo Links, Semrush’s Backlink Gap tool enables you to enter up to five domains to start comparing link profiles. It doesn’t just focus on the top links, but the data is presented based on Semrush’s Authority Rank metric (AR). And just to reiterate, Google does NOT use AR or any other third-party metric when determining rankings… It’s just an Semrush-specific metric.

After entering at least two domains, you have options for viewing the links pointing to the competition, but not your site, or vice versa. Again, it’s not perfect, but can help you start to understand powerful links leading to your competitors that you don’t have.

Note, the default view is to list the referring domains (without showing the links). To view the actual links leading to your competitor’s site, you can click the arrow to the right of the referring domain in the chart. It’s not the most intuitive way to find the links, but they are there. :)

Link Intersect in ahrefs:
Another tool that has flown somewhat under the radar is ahrefs Link Intersect Tool. It enables you to enter up to four competitor domains and view gaps in your own link profile (ordered by Domain Rank, a metric created by ahrefs). And once again, Google does not use any third-party metrics like Domain Rank, Authority Rank, Domain Authority, etc.)

Like the other tools I mentioned, the link intersect tool can shed some light on authoritative links pointing to the competition that you don’t have. For example, what are those sites linking to content-wise, do you have content like that on your own site, what can you build that’s even better than that, etc.?

To use the tool, simply enter a competitor’s domain (or several) and then your domain in the “but doesn’t link to” field. The tool will find links leading to the competition, but not to your site. The list of referring domains will be ordered by ahrefs Domain Rank. In order to view the actual links per referring domain, just click the number in the column for each competitor. You can see information about that link, including the anchor text, the url that’s linking to the site, whether the link is nofollow, if the link was found in the content, etc.

You can also export the list, but just like Majestic, you can’t export all links in one shot. You can just export the referring domains in one shot… So you’ll have to export links per referring domain, which can be tedious. So just like my note to Majestic, I have the same for Ahrefs! Please add the ability to export all links from the Link Intersect tool in one shot! That would be a great addition.

Tips and recommendations when evaluating the link authority gap:

• “Authority” is Not binary: Links (PageRank) is just one factor. Sure, it’s an important factor, but just one. Google has explained it uses a bucket of signals to determine authority. While you are analyzing the link authority gap, just understand that the links alone might not be the reason a competitor is outranking you. It’s great to see the gap, and can be important for forming a long-term strategy to improve authority-wise, but it’s not a rank-or-not-rank situation.
• Understand the Search competition versus just the competition: Make sure you are comparing to your actual competitors in Search (who you are competing within the search results) and not just which sites you think are the top competition (outside of Search). Your goal here is to understand the link authority gap between your site and sites outranking your for target queries.
• Mine the gaps, create a plan: Identify gaps in your link profile from authoritative sites, understand the content that gained those links, determine if you have content covering those topics, and figure out ways to produce killer content that can earn links like that (or even better). Don’t just try to match what your competitors are doing. Try to outdo them.
• Have patience… building authority is a long-term process: Understand that earning powerful links from authoritative sites is a long-term process. You cannot change the situation overnight… That might be frustrating, but it’s true. Understand what the competition is doing to earn those links, figure out gaps on your site, and build a killer content strategy to earn amazing links over the long-term. Once you start earning those links, you’ll appreciate that it takes time, since it can build a natural layer of protection for you (until others start going through the same process, put the work in, etc.)

Summary – Understanding the link authority gap can be enlightening, but you must execute.
By using the process I documented in this post, you can quickly start to understand the most authoritative sites linking to your competition that don’t link to your site. And that “link authority gap” can be important. But don’t stop once you uncover the gap. You need to execute based on your analysis in order to make an impact. So… what are you waiting for?? The data awaits.   

GG

When performing a competitive analysis, it’s smart to run a keyword gap analysis to determine the queries that competitors rank for that your site doesn’t rank for. It can often yield low-hanging fruit that your content team can execute on. As part of that process, it’s also smart to analyze Google’s People Also Ask (PAA) SERP feature for queries your site already ranks for, or doesn’t rank for (to determine what those additional queries are and which sites are ranking for them). I find this step is skipped often for some reason and it can also yield powerful findings that you can start to execute on quickly.

In this post, I’m going to walk you through the process of identifying ranking gaps in People Also Ask (PAA) using Semrush, which provides outstanding functionality for mining PAA data.

What we’re going to accomplish:
For this tutorial, I’m going to filter queries leading to a site by top ten rankings and then layer a secondary filter for surfacing queries where a People Also Ask module also ranks (but the site doesn’t rank in the default PAA listings). In other words, you rank in the top ten, but you don’t have content ranking in PAA for those queries for some reason. I’ve found that can yield very interesting findings that sites can execute on pretty quickly.

For example, in the screenshot below, a site ranks in the top ten for 3,359 queries when it also does not rank in the default People Also Ask (PAA) module:

Step-by-step instructions for identifying PAA gaps via Semrush:

1. First, fire up Semrush and enter the domain name you want to analyze.

2. Access the Organic Research reporting.
Click Organic Research in the left-side navigation, which will bring us to a powerful set of features for analyzing the search performance of the domain, subdomain, directory, or url you enter.

3. View all rankings for the domain via the Positions tab.
Click the Positions tab, which will yield all queries that a site ranks for in the top 100 listings (based on Semrush data).

4. Filter by top ten results:
Next, we are going to filter the results by queries where the site ranks in the top ten (so these are queries where the site ranks very well already, but might not have content that ranks in People Also Ask). I’ll cover the second part of this step next, but start by filtering by queries ranking in the top ten by clicking the Positions tab and then selecting Top 10.

5. Layer a secondary filter for PAA:
To complete this step, we want queries ranking in the top ten, but where the site doesn’t rank in People Also Ask (PAA), which can provide a great opportunity to fill gaps content-wise. To view these queries, click “SERP features”, then “Domain doesn’t rank in” and then select “People Also Ask”.

6. Analyze the data:
Now that you have two filters selected, you will be presented with all of the queries where the site ranks in the top ten but doesn’t rank in the default PAA module. When you scan the list, definitely spot-check the live search results to see which questions are listed in People Also Ask, which sites are ranking there, the content ranking for those queries, etc. Again, you can identify content gaps, format gaps (like video), and more. It’s a quick way to help your content team identify opportunities (and some gaps you find might lead you to face-palm). :)

7. Export the data:
You can always export the results if you want to use that data in other programs like Excel. You can export the data in Excel or CSV format.

And there you have it. A quick and easy way to identify ranking gaps in PAA via Semrush. It only takes a few minutes to run, and you’ll have a boatload of PAA data to go through (where your site ranks, but not in PAA). By the way, if you’re looking for other posts I’ve written about Semrush tools, then check out The Link Authority Gap which shows you how to compare the most authoritative links between websites.

PAA Gaps – Final tips and recommendations:
Before I end the tutorial, here are some final notes and recommendations based on helping clients go through this process over time. Semrush is like a Swiss Army Knife for SEO research, so make sure you are getting the most of out the tool.

• Live graphs – Remember that the graphs in Semrush are live graphs, so they change based on the filters you select. Therefore, you can see trending over time for ranking (or not ranking) in PAA when you already rank well in the organic search results. It’s a cool way to visually see your progress.
• Advanced filtering – Use advanced filtering in Semrush to fine-tune your analysis. For example, combine filters like search volume, keywords, directory, subdomain, urls, etc. You can filter the data multiple ways in Semrush (and combine those filters for advanced reporting). Play around there… you might find some great combinations that surface important data.
• PAA by country – Run this analysis by country! Just change the country you are analyzing in the reporting, and voila, you have a fresh set of queries where your site doesn’t rank in PAA.
• By device – Be sure to check both mobile and desktop data. Similar to country, just select desktop versus mobile in the filters to see each resulting dataset. You might find some differences there.
• Spot check the results – Make sure you are spot-checking the actual SERPs. PAA can obviously change (and Semrush isn’t always perfect), so make sure you really aren’t ranking in PAA for those queries. Then form a plan of attack once you identify the gaps.
• PAA formats – Keep an eye on the format of content ranking in People Also Ask. As I mentioned earlier, video could be ranking there as well. Understand the types of content Google is ranking based on query and choose the right formats for the job.
• View historical rankings – You can easily change the dates via Semrush! For example, you can look back in history and run this analysis for previous months. Have you improved, declined, or remained stable? How has Google changed with regard to PAA for those queries?

Summary: Identifying PAA Gaps Via Semrush can be powerful.
It’s hard to overlook People Also Ask when an analyzing the SERPs and the feature often contains important questions that people are searching for based on the original query. By using the process I detailed in this tutorial, you can surface and export queries where your site already ranks in the top ten search results, but doesn’t rank in PAA. In my opinion, it’s a great way to identify low-hanging fruit that your content team can dig into quickly. You never know, you might find some quick wins… or many of them. Have fun.

GG

For site migrations, I’ve always said that Murphy’s Law is real. “Anything that can go wrong, will go wrong.” You can prepare like crazy, think you have everything nailed down, only to see a migration go sideways once it launches.

That’s also why I believe that when something does go wrong (which it will), it’s more super-important to address those problems quickly and efficiently. If you can nip migration problems in the bud, you can avoid those problems becoming major issues that impact SEO. That’s why it’s important to prepare as much as you can, have all of the necessary intelligence in front of you while the migration goes live, and then move quickly to attack any problems that arise.

By the way, if you think you’re immune to site migration problems, then listen to the episode of Google’s Search Off The Record podcast where John Mueller, Gary Illyes, Cherry Prommawin, and Martin Splitt talk about the migration of the webmaster central site to the new search central site. It ends up they ran into several problems just like any other site owner could and had to move quickly to rectify those issues. So, if it can happen to Google, it sure can happen to you. :)

Adding Google’s Crawl Stats Reporting To Your Site Migration Checklist:
There are plenty of checklists and tools out there to help with site migrations. For example, Google’s testing tools in GSC, third-party crawlers like Screaming Frog, DeepCrawl, and Sitebulb, site monitoring tools, log file analysis tools, and more.

And on the topic of log files, they provide the quickest way to understand how Google is crawling your site post-migration. You don’t need to wait for data to populate in a tool, you don’t have to guess how Google is treating urls, redirects, etc., and there are several log file analysis tools hungry to consume your logs.

But there’s a catch… trying to get log files is like attempting to complete a mission as Tom Cruise in one of his great Mission Impossible movies. If gaining log files was a scene in Mission Impossible, I could hear Tom now:

“Wait, so we have to scuba dive under a bridge heavily guarded by troops, climb a 200 story building in our underwear (in 20 degree weather), use elaborate yoga moves to dodge a scattered laser security system, steal the ancient lamp of Mueller which is protected by special forces, hack into a computer system protected by six layers of ciphers, download the log files, and then parachute off the building back into the water, only to scuba dive back under the bridge to safety? No problem… hold my coffee.”

OK, it’s not that bad, but any SEO that has attempted to get log files from a client knows how frustrating that situation can be. They are huge files, seemingly not owned by one group or person in a company, and you can even find some companies not keeping logs for more than a few days (if that). So, it’s no easy feat to get a hold of them.

What’s an SEO to do?

Meet The New Crawl Stats Report in GSC: A (Pretty) Good Proxy For Log Files
In November of 2020 Google launched the new Crawl Stats reporting in GSC. The reporting is outstanding, and it was a huge improvement from the previous version. The new reporting provides a boatload of data based on Google crawling your site. I won’t go through all of the reports and data in this post, but you can check out the documentation to learn more about each of the report sections.

I’m going to cover what Google considers “site moves with url changes”, which covers domain name changes and url migrations. I’ll focus on domain name changes, but you can absolutely use the new Crawl Stats reporting to troubleshoot url migrations as well.

For domain name changes, you can view crawl stats reporting for both the domain you are moving to and the domain you are moving from. So, using the crawl stats reporting can supplement your current migration checks and enable you to see how Google is handling the migration at the source (the old domain).

And for url migrations, you can also surface problems that Google is experiencing post-migration. It’s not as clear as a domain name change, since you can’t isolate the crawl stats reporting by domain, but it can still help you surface issues based on bulk-changing urls.

Note: There is a delay in the Crawl Stats reporting.
The Crawl Stats reporting lags by a few days, so log files are still important if you want to see a real-time view of how Google is handling a site migration. The reporting updates daily, but lags by 3-4 days from what I have seen. Below, you can see the report was last updated on 2/26/21, but today is 3/2/21.

How to identify problems with domain name changes and url migrations using the Crawl Stats reporting in GSC:
As mentioned above, for domain name changes, you can analyze the Crawl Stats reporting for the domain name you are moving from, and the domain name you are moving to. Below, I’ll cover some of the ways you can use the reporting to surface potential issues.

How To Find The New Crawl Stats Report in Google Search Console (GSC):
First, I know there’s some confusion about where the new Crawl Stats report is located. You will not find the report in the left-side navigation in GSC. Instead, you first need to click “Settings”, find the Crawling section of the page which contains top-level crawl stats, and then click “Open Report” to view the full Crawl Stats reporting.

Now that you’ve found the Crawl Stats reporting, here are some of the things you can find when analyzing and troubleshooting a site migration.

404s and Broken Redirects:
The Crawl Stats reporting for the domain you are moving from will list urls that Google is crawling that end up as 404s. All urls during a domain name change should map to their equivalent url on the new domain (via 301 redirects). By analyzing the source domain name that’s part of the migration, you can view urls that Googlebot is coming across that end up as 404s. And that can help you find gaps in your 301 redirection plan.

For example, you can see the reporting for a site that went through a domain name change below. 4% of the crawl requests were ending up as 404s when most of the urls should be redirecting to urls that return a 200 header response code on the new domain.

If you click into that report, you can see a sample of the top 1,000 urls with that issue and you can inspect the urls as well:

And here is what it should look like. 100% of the requests are 301 redirecting to the equivalent urls on the new domain:

Important (and often confusing) note: It’s worth noting that GSC reports on the destination url, so a 404 showing up for the old domain name could actually be showing you a redirect to the new domain name, but that new url 404s. In other words, the 404 is actually on the new domain, but shows up in the reporting for the old domain name. That’s extremely important to understand overall with GSC, and it can cause confusion while analyzing the reporting. I tweeted about this in January with regard to the Coverage reporting:

Image Search: Googlebot Image
If image search is important for your business, you will definitely want to review the “By Googlebot type” section of the reporting. You will see a listing for “Image”. You can click into that reporting to see the urls Googlebot Image is crawling. If you see 404s, 5XX, etc., then make sure you jump on those issues quickly. You should see plenty of 301s if you redirected images properly during the migration (which you should). I covered that in my Mythbusting video with Google’s Martin Splitt about site migrations. The video can be seen later in this post.

As you can see below, Googlebot Image is coming across 404s as well. This is from the site that went through a domain name change.

This is what you should see. Notice how the Googlebot Image requests all properly 301 redirect to the images on the new domain:

Robots.txt issues:
In the host issues section, you can see if Googlebot is having problems accessing the robots.txt file for the domain(s) involved. If Google cannot fetch the robots.txt file (which returns a 200 or 403/404/410), then it will not crawl the site at that time. Google will check back later to see if it can fetch the robots.txt file. If it can, then crawling will resume. You can read more details about how this is handled on Google’s support page (or in the screenshot below). Note, you can 404 a robots.txt file and that’s absolutely fine. This is about Google having problems fetching the file (i.e. Google seeing a 429, 5XX).

And here is what it can look like in GSC’s Crawl Stats reporting. Although this falls under an “acceptable fail rate”, I would sure check why the robots.txt fetch is failing at all:

Other host issues: DNS Resolution and Server Connectivity:
Along the same lines, you can see if there are other host-level issues going on. The host reporting also contains DNS resolution errors and server connectivity problems. You obviously want to make sure Google can successfully recognize your hostname and that it can connect to your site.

Performance Problems:
The reporting also will show pages that are timing out for some reason, so keep an eye on that report. You will find that in the “By response” section.

Subdomain Issues:
Hopefully you picked up all subdomains that were in use before you pulled the trigger on the domain name change. But if you didn’t, you can see Crawl Stats reporting per subdomain that Google is crawling. The catch is that you need a domain property set up in GSC for the domain you are moving from (unless you had those subdomains verified and set up already in GSC).

If you did, you could view the crawl stats reporting for those subdomains separately. Domain properties make this easier since all subdomains being crawled by Google (the top 20 over the past 90 days) will be shown in the Hosts report in the Crawl Stats reporting.

Below, you can see that the crawl stats reporting shows 17 different subdomains with crawl requests over the past 90 days.

Note, I always recommend having a domain property set up. It’s amazing how many companies have not done this yet… If you haven’t, I would do that today. It doesn’t take long to set up and it covers all protocols and subdomains.

Crawl Stats For Site Migrations: Final tips And Recommendations
The Crawl Stats reporting can help site owners and SEOs get closer to log file analysis, when gaining those logs might be tough. Although there’s a lag in the data populating (3-4 days), the Crawl Stats reporting can sure help surface problems during domain name changes and url migrations. And the quicker you can nip those problems in the bud, the less chance they become bigger issues SEO-wise.

Here are some final tips and recommendations:

• Set up domain properties for each of the domains involved in the migration (if changing domain names). This will give you access to all subdomains in the Crawl Stats reporting.
• Once data starts populating in the Crawl Stats reporting post-migration, dig into the domain you are moving from. You might see a number of issues there based on what I explained earlier. For example, 404s, performance issues, robots.txt problems, and more.
• Nail the redirection plan. If you see gaps and problems with your 301 redirects, move quickly to rectify those problems. Nip those problems in the bud.
• Check for host-level problems (like robots.txt fetch issues, DNS resolution issues, and server connectivity problems). Your redirection plan doesn’t matter if Google can’t successfully connect to your site.
• Look for pages that are timing out. This would show up in the “By response” section of the reporting. If you see that, dig into those problems to see why the pages are timing out. Again, move quickly to address performance issues.
• Don’t forget your images! Make sure to 301 redirect your images and then check the section labeled “By Googlebot type”. Then check the “Image” reporting to see how Googlebot Image is crawling your content.

More About Site Migrations: Mythbusting Video
If you are interested in site migrations, then you should check out the Mythbusting video I shot with Google’s Martin Splitt. In the video, we cover a number of important topics including domain name changes, url migrations, redirecting images, when a site should revert a migration, site merges, the Change of Address Tool in GSC, and more.

Summary – GSC’s Crawl Stats as a proxy for log files.
After reading this post, I hope you see the power in adding Google’s Crawl Stats reporting to your site migration checklist. The reporting provides a boatload of great information based on Google crawling your site post-migration. I’ve found it extremely helpful while helping companies monitor and troubleshoot domain name changes, url migrations, and more. And remember, Murphy’s Law is real for site migrations. Things will go wrong… which is ok. The important part is how quickly you handle and rectify those problems.

GG

On January 11, 2021 Google announced improvements to GSC’s Index Coverage data. With the update, which is clearly apparent in the reporting, Search Console removed the generic “crawl anomaly” issue (and mapped those to more granular issues), changed how urls are reported that are blocked by robots.txt, added a new issue called “indexed without content” as a warning, and then had one final note about soft 404s. That last bullet said, “Soft 404 reporting is now more accurate.” And based on my work with large-scale sites, I was particularly interested in that final bullet.

A number of the sites I help had tens of thousands, or even hundreds of thousands of soft 404s being reported based on the nature of their content. Starting on January 5, 2021, the number of soft 404s reported in GSC fell off a cliff. I’m talking about a severe drop, like these examples:

In order to understand why alarms were going off in my office, and why I’m particularly interested in those drops, let me quickly cover more about soft 404s. Then I’ll explain what I found once I dug into each situation.

A Deeper Look at Soft 404s:
Google explains that a soft 404 is a url that tells the user the page doesn’t exist, but it still returns a 200 header response code (instead of a hard 404). In practice, the soft 404s that Google has surfaced historically have been urls that span different page types. For example, the most obvious is a product that’s no longer available that still returns a 200 response code. It might have very little content and a message telling users the product is not available on the site anymore.

But, other types of pages have showed up in the soft 404s reporting over time. For example, I’ve seen super-thin pages show up there like articles, slideshows, job listings, pages with images or video with little textual content, and more. And on larger-scale sites, those pages can be problematic quality-wise depending on how Google is handling them.

For example, a soft 404 is treated as a hard 404. Google will not index those urls identified as soft 404s. But, if a site had a big soft 404 problem, I often found that Google wasn’t always handling all of those pages types as soft 404s… And that led to many thinner urls getting indexed.

So, if you had 50K super-thin pages and Google saw them as soft 404s, then technically that shouldn’t be a huge problem quality-wise. Since they are seen as hard 404s, Google will obviously not take them into account when evaluating quality.

But, what if suddenly those urls aren’t seen as soft 404s anymore? Well, those urls could be counted when Google evaluates quality overall for the site. That’s a big reason my “quality antennae” went up when I saw tens of thousands, or even hundreds of thousands, of soft 404s suddenly disappear. Are they indexed now, categorized as another issue in GSC and not being indexed, or maybe not even known to Google anymore? I NEEDED TO KNOW!

In addition, soft 404s can impact crawl budget, which can be a concern for larger-scale sites. Note, most sites do not need to worry about crawl budget. Only very large sites (typically sites with over one million urls) need to worry about crawl budget. For example, this site needs to worry about crawl budget:

Google’s own documentation explains that soft 404s can eat up crawl budget. So it’s another concern when it comes to soft 404s (and understanding where they magically went on 1/5/21). In the past, if you saw them flagged in GSC, then you could identify patterns and handle appropriately. But if they suddenly disappear, it could be harder to surface and handle.

In Search Of: Lost Soft 404s
This all leads me back to one question… where did all of those soft 404s go?? After seeing the number of soft 404s drop off a cliff on 1/5/21, I decided to dig into the data across sites and find out what happened. So, I grabbed my virtual pickaxe and headed into the GSC mine. I was determined to find out how those urls were being treated now.

Now, if you choose to dig into search console after a change is rolled out by the GSC product team, you’ll quickly realize that you cannot access historical data. That sucks and means you will have a hard time finding urls previously categorized as soft 404s unless you documented them somehow. I had this covered since I always dig heavily into GSC data when auditing client sites. I was able to open my vault of recent audit documents and resurface many soft 404s that were reported over the past six to twelve months.

Based on digging into the data, I was sometimes relieved with what I discovered for certain situations, but also alarmed by what I found in other situations. I’ll explain more below.

Where Did Soft 404s Go and How Is Google Currently Handling Those URLs?
Below, I’ll cover the various categories of findings based on digging into a number of large-scale sites that previously reported many soft 404s. On 1/5/21, the number of soft 404s dropped off a cliff and I was determined to find out how they were currently being treated.

Super-interesting: “The URL is unknown to Google”
This was incredibly interesting to me. I noticed many url across sites that were once reported as soft 404s now returning “The URL is unknown to Google” when inspected in GSC. It’s almost like Google completely wiped out the history for these urls.

This isn’t a category that shows up in the Coverage reporting for obvious reasons… since it’s NOT known to Google. But, it was known to Google at some point recently. I know that for sure. Google’s documentation does explain more about this categorization, but I’m not sure it answers why the soft 404s are now “unknown to Google”. Here is what Google’s documentation explains:

“If the index coverage status “URL is unknown to Google” appears, it means that Google hasn’t indexed the URL either because it hasn’t seen the URL before, or because it has found it as a properly marked alternate page, but it can’t be crawled.”

I’m going to reach out to Google to learn more about this. I would imagine that Google handling many urls as this category is helping it improve the efficiency of its own systems, but again, I’ll see if Google can comment about this. But if it’s unknown to Google, it won’t hurt your site quality-wise. That’s good.

“Crawled, not indexed” and “Discovered, not indexed”:
I have covered this category a number of times in my other posts about technical SEO and broad core updates. Both “Crawled, not indexed” and “Discovered, not indexed” can signal quality problems and/or crawl budget issues. With “Crawled, not indexed”, Google discovered and crawled the url, but has chosen to not index it. I have often found low-quality or thin content when analyzing this category across sites (or just parts of a site Google isn’t digging for some reason).

So, if your soft 404s end up as “Crawled, not indexed”, then that’s ok in the short-term (since they aren’t being indexed and can’t be held against the site quality-wise), but I would still heavily dig into the urls and handle them appropriately. This is also one of the reasons I believe we need larger exports from GSC’s Coverage reporting! Exporting this category in bulk would help site owners identify more patterns across the site that could be causing problems.

And for “Discovered, not indexed”, Google knows about the url, but has decided to not even crawl it (and clearly not index it). I have seen quality problems and problematic url patterns while analyzing this category of urls. For example, with some large and complex sites with deep quality issues, it’s like Google knows ahead of time what it’s going to find on many of those urls (based on url structure) and doesn’t even want to crawl them.

So, if you see many soft 404s now being categorized as “Discovered, not indexed”, then that’s also ok in the short-term since they can’t be held against the site quality-wise if they aren’t being indexed. But just like with “Crawled, not indexed”, I would dig in and make sure the urls are being handled properly. If they should 404, then make sure they return hard 404s. If they are thin and low-quality, then boost the content there. And if it’s a problematic area of the site, then work on improving that area overall. Remember, “Discovered, not indexed” means Google knows about the urls, but doesn’t believe it’s worth it to even crawl them (let alone index them).

URLs Now Indexed: Danger Will Robinson! Danger!
Yes, this is a super important category. When digging into urls that were previously categorized as soft 404s, I noticed some of them were now being indexed! That can be a dangerous situation for some sites. For example, if they were thin and low-quality urls that Google is now indexing for some reason, those urls will now be taken into account while Google evaluates quality for the site overall. “Quality indexing” is very important (especially for larger-scale sites), so having an influx of low-quality content get indexed is not a good thing.

If you notice many urls that were once flagged as soft 404s now being indexed, then it’s important to handle those urls appropriately now. For example, you can noindex low-quality content, boost the content there (improve it so it meets or exceeds user expectations), 301 it to a newer version of the content, or 404 the pages if they shouldn’t be on the site anymore. But I would not leave them as-is if they are low-quality or thin.

True soft 404s still seen as soft 404s:
Even though many soft 404s disappeared from the reporting in GSC, there were still urls categorized as true soft 404s. For example, pages that were literally blank that returned 200 header response codes, pages that returned “not found” messages on the page, but returned 200 response codes, etc. Again, true soft 404s. I guess this is the “handling soft 404s more accurately” part from Google. For these urls, I would make sure to handle them appropriately. If they should 404, then have them return hard 404s and not 200 header response codes. And if they shouldn’t be seen as soft 404s, then fix or boost the content there.

Moving Forward: What Site Owners Can Do
If you have seen a big drop in soft 404s starting in early January 2021, then I highly recommend digging into the situation. If you don’t have historical data containing urls that were once categorized as soft 404s, there’s not really much you can do to see how those soft 404s are being reported now. But, if you did document soft 404s for your site, then you can inspect those urls to see how Google is handling them now.

Are they “Unknown to Google” now, are they being categorized as “Crawled, not indexed”, “Discovered, not indexed”, are they still soft 404s, or are they being indexed? And if they are being indexed, then they can be counted when Google evaluates quality for the site overall. That’s why it’s important to handle the urls appropriately.

Summary – Know Your Soft 404s
With the latest data improvements in GSC, Google started better handling soft 404s. That’s great, but many sites saw a big drop in urls reported as soft 404s at that time. Since that category in GSC often yielded thin or low-quality content on a site, it’s important to understand how Google is currently handling the urls. Are they still not being indexed, or are they now being indexed and counting towards quality? I recommend digging into the situation for your site and then handling those urls appropriately.

GG