The Internet Marketing Driver

The March 12, 2019 Google Core Algorithm Update – A Softer Side Of Medic, Trust And The Link Graph, Quality Still Matters, And The Importance of the “Kitchen Sink”

By Leave a Comment

In 2018, we saw three broad core ranking updates that caused massive volatility in the search results globally. Those updates rolled out in March of 2018, August of 2018 (Medic Update), and then late September of 2018. All three were huge updates, which sent some sites dropping off a cliff and others surging through the roof.

Since late September/early October of 2018, we have been waiting for the next big core update. And it finally arrived on March 12, 2019. And the update didn’t disappoint. Once again, there was a ton of movement across sites, categories, and countries. It didn’t take long to see the impact. For example, here is a massive surge in search visibility and a large drop all starting on March 12.

A site that surged during the March 12 core update.
A site dropping off a cliff after the March 12 core update.

Although there was a lot of movement (and chatter) about the health/medical niche again, the update clearly impacted many other categories (just like the Medic update did). For example, I have many sites documented that saw significant movement in e-commerce, news publishers, lyrics, coupons, games, how-to, and more.

Note, I cover a number of topics in this post. To help you navigate to each section, I’ve provided a table of contents below:

Google’s comments about the March 12 Core Update: Reversals, Neural Matching, and Penguin
Before I get into my analysis, it’s important to know that Google provided some information about the March 12 update to Barry Schwartz at Search Engine Land. Google explained that the latest update on 3/12 wasn’t a full reversal. That makes complete sense based on what I’m seeing. More on that soon.

Google also explained that all of the recent core updates (including this one) had nothing to do with any neural matching updates. They checked each of those updates and NONE lined up with these core updates. That’s incredibly important to understand since there are some that believe the updates had a lot to do with neural matching, which is an artificial intelligence method designed to help Google connect words to concepts.

Google’s comments about neural matching and core updates.

The example they provided last fall when they mentioned neural matching demonstrated that you could search for what the soap opera effect was by describing it in your query, and Google could know what you were referring to. If you want to learn more about neural matching, Google just provided information about the differences between neural matching and RankBrain

Google also explained that these updates had nothing to do with Penguin. Now, that doesn’t mean Google isn’t evaluating links as part of this update… it just means that Google’s Penguin algorithm didn’t have anything to do with the March 12 update. More about links soon.

A Full Medic Reversal! Not so fast…
Based on the impact, it was easy to see that many sites that were impacted by the August update (Medic) saw a change in direction. And some saw a radical change in direction, which led some to believe that the March 12 update was a complete reversal of Medic. That’s definitely not true based on the data I’ve been analyzing (and what many others have seen as well).

For example, there are definitely sites that surged on March 12 that dropped heavily on August 1, but there are some sites that dropped more on March 12 that dropped heavily on August 1. And then there are some sites that surged more on March 12 that increased on August 1. Here is an example of a site that reversed course after getting hit by the Medic update:

Another sites seeing recovery after a Medic hit.

But, here are two sites seeing more movement in the same direction (they were either hit by Medic and dropping more on March 12, or surged during Medic and then gained more).

A site dropping more during the March 12 update after a Medic hit.
A site surging more on March 12 after a Medic surge.

Also, it’s worth noting that there are many sites that saw some improvement on March 12 that had dropped heavily on August 1 (so they experienced a partial recovery and not a full one). And on the flip side, there are some sites that dropped on March 12 after surging in August, but didn’t drop all the way back down to their previous levels. For example, draxe.com surged back on March 12, but not all the way back. There are many sites with trending like that:

draxe.com reversing course on March 12 after a big Medic drop.

Side note: Since I’ve been analyzing the health/medical niche heavily since the August update, it was ultra-interesting to see “the big three” get knocked down a few notches. I’m referring to healthline.com, webmd.com, and verywellhealth.com. In particular, healthline.com and verywellhealth.com experienced some big drops in search visibility. That makes sense, since many other sites in health/medical surged during this update. Based on those surges, it’s only natural that the top three players who were dominating health queries would experience a drop as others see gains. But there may be more to those drops than just that. I’ll cover more about what I’m seeing soon after analyzing many sites that saw movement during this update.

The big 3 sites in health dropped during the March 12 update.
For example, verywellhealth.com lost significant search visibility.

Algorithm Tinkering – Softening From The Medic Update & E-A-T Calculations
I mentioned the idea of a Medic reversal earlier in the post and how I don’t believe there was a full reversal. But that doesn’t mean Google couldn’t have softened some of the algorithms it used in the August update. Again, there were many sites seeing improvement during the March update that got hammered during the August update.

I was able to ask John Mueller this exact question during a recent webmaster hangout video. John gave a vague answer, but he did explain that Google sometimes goes too far with an update and needs to pull it back. He also said this can work the other way around, where Google didn’t go as far as they should, so they strengthen their algos.

Here’s the video of John explaining this (at 20:04 in the video):

A softening of the Medic Update is entirely possible. It was one of the biggest and baddest updates I have ever seen. There were extreme drops and surges in traffic everywhere I looked. And the health/medical niche saw the most volatility (hence the name Medic).

Every time I analyzed a site impacted by the August update, I couldn’t help but think the update had a unique and extreme feel. Some sites got slaughtered when it was hard to see why they would deserve dropping that much (70%+ for some sites). Anyway, the important part of this section is that Google could have tinkered with its algorithms to pull back some of the power of the Medic update. That’s entirely possible and could cause massive volatility when they do that.

That’s a good segue into the most scalable way for Google to tinker with trust, especially for Your Money or Your Life (YMYL) sites and content – I’m referring to links.

The Link Graph and Tinkering With Trust
After analyzing the Medic update, and then the September update, I explained that the best way for Google tinker with trust would be via the link graph. For example, if Google simply increased the amount of power of certain links, while decreasing the power of other links, then that could cause mass-volatility across the web (just like we saw with the August update).

And that’s especially the case with YMYL sites. For example, queries that can “impact the future happiness, health, financial stability, or safety of users”. You can read Google’s Quality Rater Guidelines (QRG) for more information about YMYL sites.

After the Medic update, it was clear that YMYL sites were impacted heavily, and more heavily than other types of sites. After analyzing a number of sites and digging into their link profiles, you could see the gaps between certain sites that surged and dropped (from a link power perspective). For example, I found links from the CDC, CNN, and other extremely powerful domains for sites that surged when comparing to competing sites that dropped.

Let’s face it, you can’t just go out and gain links like that overnight. It led me to believe that Google could be tinkering with trust via the link graph. I included information about that in my post about the September 27, 2018 core update and in a separate LinkedIn post.

Beyond what I just explained, Google released a whitepaper recently that made this even clearer. That whitepaper revealed more information about how Google evaluates expertise, authoritativeness, and trust (E-A-T) algorithmically. And it supports my point about tinkering with trust via links.

Google explained that E-A-T is algorithmically evaluated by several factors, the best-known factor being PageRank (which uses links across the web to understand authoritativeness). Yes, that means Google can use links to evaluate E-A-T, which makes complete sense. Like I’ve said in the past, it’s the most scalable way to tinker with trust…

Going one step further, Google also explained that when it detects YMYL queries, it can give E-A-T more weight. So, that means the right links can mean even more when a query is detected as Your Money or Your Life (YMYL). This could very well be why many YMYL sites were heavily impacted by the August update (Medic). It’s just a theory, but again, makes a lot of sense.

Now with the March 12 update, we very could be seeing an adjustment to the algorithms that are used to evaluate E-A-T. Any sort of power adjustment to specific links can cause mass volatility (especially for YMYL queries). Remember, many YMYL sites saw a lot of movement during the March 2019 update, so it’s entirely possible. And as I explained above, Google’s John Mueller said that Google can dial-down certain algorithms if it believes it went too far.

Examples of Surges and Drops
I have a list of 165 domains that were impacted by the March 12 update, including a number of clients that saw movement (either up or down). I wanted to provide some examples of impact (both positive and negative), including findings based on those surges or drops. I can’t cover everything I’ve seen while analyzing sites that were impacted, but I did want to cover some extremely interesting situations.

Disclaimer: Before we begin, I have to provide my typical disclaimer regarding major algorithm updates. I do not work for Google. I don’t have access to its core ranking algorithm. I don’t have a hidden camera in the search engineers’ cafeteria in Mt. View, and I haven’t infiltrated Google headquarters like Tom Cruise in Mission Impossible. I have been to Google Headquarters in both California and New York, and it was tempting to crawl through the vents to find a computer holding all of Google’s algorithms, but I held back. :)

I’m just explaining what I’ve seen after analyzing many sites impacted by algorithm updates over time, and how that compares with what I’m seeing with the latest Google updates. I have access to a lot of data across sites, categories, and countries, while also having a number of sites reach out to me for help after seeing movement from these updates. Again, nobody knows exactly what Google is refining, other than the search engineers themselves.

Sites that surged:

Health/Medical Site – Taking The “Kitchen Sink” Approach To Remediation
The first site I’ll cover is in the health/medical niche that got hit hard during the Medic update in August of 2018. The site lost over 40% of its Google organic traffic overnight.

Once I dug into the situation, there were a number of problems I was surfacing. I worked with this client for over four months on surfacing all potential problems across the site, including content quality problems, user experience (UX) issues, aggressive monetization, a lack of author expertise for certain types of content, render and performance problems and more.

The site owners have worked hard to make as many changes as possible during the engagement, and they plan to tackle even more of them over time. Overall, a number of important changes were made to improve the quality of the site, increase E-A-T as much as they could, fix all technical SEO problems they could, decrease aggressive monetization, etc.

And then March 12 arrived, and the site began to surge. As of today, it’s up 72% since 3/13 (the first full day of the rollout). It’s a great example of taking a “kitchen sink” approach to remediation. Now, as mentioned earlier, it’s hard to say how much of the surge is based on remediation versus a softening of what rolled out in August (or some combination of both), but it’s hard to ignore all of the changes this site made since then.

E-commerce site surging – The Importance of Being Proactive Versus Reactive
Another client that surged is an e-commerce retailer selling a high-end line of products. I helped them a few years ago during medieval Panda days. They improved greatly during that time and their organic search traffic surged during 2015 and 2016. And they contacted me a few months ago after seeing declines during both the August and September updates in 2018. Unfortunately, the site had gone off the tracks slightly and they were experiencing a downturn in traffic.

So I dug into a crawl analysis and audit of the site and began surfacing any problems that could be hurting them. That included digging into their technical SEO setup, reviewing content quality, user experience (UX), performance, and more.

They tend to move fast with changes, so as I was sending findings through, they were addressing those problems very quickly. For example, there was a massive canonicalization issue across many of their category and product pages. In addition, there were some content problems riddling the site.

In particular, all of their category pages had the dreaded fluff descriptions that some e-commerce retailers provide. You know, forced content that doesn’t really help users… And to add insult to injury, most of those descriptions (which were very long), were partially hidden on the page. Users had to click “read more” to reveal the full description. It was pointless and was clearly there just for SEO purposes.

Even though John Mueller has repeatedly explained that e-commerce sites should NOT do this, and that they don’t need to do this, many still do employ this tactic. My client took a leap of faith and removed most of that content from every category page on the site (many). They replaced it with just a few helpful lines of copy that was crafted for the user. So about 80% of the text was removed from each description.

In total, they made a number of important changes on the site (beyond what I just explained). And on March 12, they began to surge. It was fascinating to watch a number of those category pages jump to #1 for their most competitive queries. Not #2 or #3…. But #1. And their average sale is in the thousands of dollars. Needless to say, they were excited to watch this happen. The site is up 57% since 3/13 (the first full day of the rollout).

An e-commerce category page jumps to #1 after the March 12 update.

B2C Content Site (including information and reviews) – A Competitive Battle
The next example is a battle between two competitors. I’ve helped one of the sites extensively that has experienced a decline over time. They have worked hard on improving the site overall, including nuking a lot of lower quality and thin content from the site, while also improving the user experience. They have experts producing content and have built a very strong brand over the years. Actually, it’s one of the strongest brands in its niche.

The other site has surged over the past few years and had overtaken my client in terms of search visibility. They are a relatively new brand in the space (think years versus decades for my client). Even though they have surged over the past two years, they have many problems across the site from a quality standpoint, which includes aggressive advertising problems, thin content, over-optimization, and more.

Starting in August, the competitor finally started to drop after years of increasing. And during the March 12 update, they dropped even more. Both sites have insanely powerful link profiles with millions of links each (and many from powerful sites in their niche). But my client has cleaned up many problematic things over the past year, while the competitor has all sorts of problems as mentioned above.

E-commerce retailer in a controversial niche – BBB vs. overall reputation
The next example is an interesting one, considering the recent focus on E-A-T. It’s an e-commerce retailer that’s in a controversial niche. It got smoked during the Medic update, almost getting cut in half visibility-wise. The site has an e-commerce store, but also a lot of educational information, tips, etc. There are thousands of user reviews for the site, with a high average rating. The reviews are handled by a third-party service.

During the latest update on March 12, the site absolutely surged. It has regained a good amount of visibility, although it’s not back to where it was prior to the Medic update.

But there’s a reviews dichotomy here, which is interesting. Their user reviews are very strong, but their BBB profile is not. They have an F rating and there are a number of complaints that haven’t been addressed. I’ll touch on BBB ratings later as well, but it’s a great example of Google not using the hard BBB rating, but possibly evaluating reviews from across the web (which makes much more sense).

How-To Content (UGC) – Managing “Quality Indexation”, Improving Technical SEO, and Subdomain Impact
The next site is a large-scale site focused on how-to content that dropped during the Medic update. Since there’s a lot of user-generated content (UGC), there is always the danger of thin or low-quality content getting published at scale. The site has several subdomains targeting different countries.

Once digging into a crawl analysis and audit of the site, I surfaced many different issues across a number of important categories. For example, although there was a lot of 10X content (super high-quality), there was a lot of thin content and low-quality content mixed in. My client moved quickly to determine which pieces of content should be nuked from the site (by either 404ing or noindexing content). They have removed about 100K urls as of today.

Next, there were technical SEO issues that could be causing quality problems. For example, canonical issues, performance problems, some render issues, and more. This client moves very quickly, since their dev team is great. It’s not uncommon for me to send findings through and have them implement changes within a few days (or even quicker).

With the March 12 update, we initially thought the site didn’t see much of a gain. But the devil’s in the details. When you look at overall traffic or search visibility, the site didn’t move very much. But, when you check each subdomain, the trending tells a different story. The international subdomains experienced nice gains during the update, while other pieces of the site either remained stable (or even decreased slightly). It’s a great example of a broad core ranking update impacting sites at the hostname level. More about that next. Here’s how some of the subdomains looked after the March 12 update:

Impact at the hostname level. Subdomains increasing during the March 12 update.
Another subdomain surging during the March 12 update.

Side note about hostname-level impact: Reminder, broad core ranking updates can impact sites at the host level (like the subdomain example I just provided). If you’ve followed medieval Panda closely, then that should sound familiar. Here’s a tweet from Pedro Diaz explaining a conversation he had with Google’s Gary Illyes. Gary explained that algorithms like Panda collect data at the page-level, but act at the hostname-level. That’s exactly what I’ve seen across a number of sites that were impacted by these broad core ranking updates over time. It’s just an interesting side-note:

Examples of sites that dropped:
Large-scale YMYL Health/Medical – Being Proactive After Surging With Every Update Until March 12
The first example of a big drop is a YMYL site that had been doing extremely well in a very competitive niche. They reached out to me in the fall since they wanted to have the site audited to make sure they could avoid damage down the line. In other words, although they were surging, they didn’t really know why. And they wanted to make sure they continued doing well. So, they were proactively seeking SEO assistance versus reactively addressing a hit down the line. That’s always a smart approach.

Health site surging during the Medic update.

I started analyzing the site in February and it didn’t take long to surface some very big problems. With every twist and turn, I was finding many issues including massive thin content problems, low-quality content, JavaScript render problems, canonical issues, and more. I literally couldn’t figure out why this site had been surging so much.

I brought this to my client’s attention very quickly and have reinforced that point several times. They did begin making some changes, but I mentioned that the next update could bring bad news (since I had a good feeling we were close to a big update… since we hadn’t seen a broad core ranking update since September of 2018).

And then March 12 arrived and the site got hit hard. As of today, the site is down 37% since March 13 (the first full day of the rollout). And when checking queries that dropped and their corresponding landing pages, they line up with the problems I have been surfacing. For example, thin content, empty pages, pages that had render issues, so on and so forth.

The site got hit hard during the March 12 update.

Large-scale Health Publisher – no E-A-T, republishing content, aggressive ads
The next example is a YMYL site focuses heavily on health. It surged during the Medic Update and then even more during the September update.

But it just got crushed during the March 12 update. There are a number of problems across the site that are hard to ignore. First, much of the content has no author listed, so it’s impossible to know if it’s written by a doctor or a kid in high school. Author expertise is extremely important, especially for YMYL content.

Second, there’s an aggressive ad problem. There are low-quality ads all over the site, especially mega-loaded at the bottom of the page.

It’s also worth noting that the site uses an m-dot for its mobile content. With mobile-first indexing, it’s extremely important to make sure your m-dot contains the same content, directives, structured data, links, etc. as your desktop version. The mobile version is what Google is using for indexing and ranking purposes. There were definitely gaps between the desktop pages and mobile pages, so that could be contributing to the drop.

For example, here are some problems I surfaced when comparing mobile pages to desktop pages:

Also, and this is important, the site consumes a lot of syndicated content. I’ve mentioned problems with doing this on a large scale before and it seems this could be hurting the site now. Many articles are not original, yet they are published on this site with self-referencing canonical tags (basically telling Google this is the canonical version). I see close to 2K articles on the site that were republished from other sources.

Here is what happened during the March 12 update. The site lost a significant amount of search visibility:

Niche Publisher (International site) – Aggressive and Deceptive Ads, Not Secure, Lacking Author Expertise
The next site I’ll cover is an international publisher focused on a very specific niche. The site was not affected by the Medic update in August, but surged like mad during the September update.

Publisher surging during the September 2018 update.

They increased even more since September and had surpassed 80K users per day. But March 12 ended up being a very bad day for them. The site dropped heavily losing 46% of its Google organic traffic since the update rolled out.

When reviewing the site, there were a few things that stood out immediately. First, there was a big aggressive advertising problem (with certain elements that were deceptive too). For example, the hero image for each article was a large ad right in a core area of the content. I’ll guarantee some users were clicking that image and being whisked off the site to the advertiser site. I’ve mentioned ads like this before many times in my articles about major algorithm updates. Hell hath no fury like a user scorned.

Next, there were aggressive ads weaved into the content. For example, accordion ads that were injected in between paragraphs. They were large and intrusive.

Beyond that, there is no author information at all. For each article published, users (and Google) have no idea who wrote the content. With Google always wanting to make sure they are sending users to the most authoritative posts written by people with expertise in a niche, having no author information is not a good idea.

And last, but not least, the site still hadn’t moved to https. Now, https is a lightweight ranking factor, but it can be the tiebreaker when two pages are competing for a spot in the SERPs. Also, http sites can turn off users, especially with the way Chrome (and other browsers) are flagging them. For example, there’s a “not secure” label in the browser. And Google can pick up on user happiness over time in a number of ways (which can indirectly impact a site rankings-wise). Maybe users leave quickly, maybe they aren’t as apt to link to the site, share it on social media, etc. So not moving to https can be hurting the site on multiple levels (directly and indirectly).

Large-scale Lyrics Website – UX Barriers, Aggressive, Disruptive, and Deceptive Ads + The Same Content As Many Other Sites
The next site I’ll cover is a large-scale lyrics site that has gotten hit by multiple algorithm updates over the years (including when medieval Panda roamed the web). And it had experienced significant volatility in 2018, where it dropped during the March 7, 2018 update, and then surged with the September 2018 update. It was clearly in the gray area of Google’s algorithms. Unfortunately, the site just got hit hard by the March 12 update, cutting some of the gains it made in September of 2018.

When reviewing the site, it was clear there were some problems. First, any site that provides the same exact content as other sites must provide some type of value-add. If not, you are leaving Google with a very hard decision when it’s presented with many options for users searching for the same content available on many sites across the web.

Actually, Google’s John Mueller just covered this again during a recent webmaster hangout video. He explained that if a site contains the same exact content as many others, then they should try to provide as much unique value as possible.

Here is the video (at 45:10 in the video):

This site does not provide a value-add. It’s literally the same lyrics content that’s available on many other sites. As an alternative, some sites provide song meanings, information about the artists, concert information, and more.

Second, there is ultra-aggressive advertising on the site. I’ve mentioned many times in the past the problems with aggressive and disruptive advertising and how sites that employ them often fare during major Google algorithm updates (especially when combined with other problems). This site contained deceptive ads in prominent areas of the content, which I’m sure is infuriating some users.

So, we have a site with aggressive, disruptive, and deceptive advertising, and the same exact content as many other sites on the web (without providing any value-add). The combination is clearly not working for them (as they have ridden the Google roller coaster for years – surging and dropping with many Google algorithm updates).

A Note About Reputation – Focus on overall reputation, not a single score.
I don’t want to spend too much time on this, since this post is already long. We know that Google doesn’t use hard BBB ratings when evaluating sites. I asked John Mueller about that a few months ago. But, that doesn’t mean Google is ignoring overall reputation across the web. There’s a big difference between the two.

After the March 12 update, I checked the ratings and reviews for a number of sites that surged or dropped and found that BBB ratings/reviews often did not correlate with surges or drops (meaning strong BBB ratings would lead to a surge and weak BBB ratings would lead to a decline). That said, overall reputation could be impacting those sites.

For example, I noticed that a large gaming company has a BBB rating of F with many complaints. They didn’t drop at all during this update. Overall, their games have been a massive hit with millions of users. And many of those users love the game (and have reviewed it, blogged about it, shared that across social media, and more). So, it’s a good example of site with an F rating from the BBB, but a different reputation across the entire web.

I also checked a huge e-commerce retailer, which surged during the March 12 update, even though it has 510 complaints via the BBB. But they have a huge following and a much different reputation overall across the web. This also leads me to believe that if Google is using reputation, they are doing so in aggregate and not using third-party scores or ratings.

This topic could yield an entire post… but I just wanted to mention reputation as it relates to these broad core ranking updates. In my opinion, I would focus on overall reputation (which is what you should be doing anyway). And even if you have specific third-party ratings that are low, that doesn’t mean those specific ratings will drag your site down if you have other positive signals across the web.

What Site Owners Can Do – The “Kitchen Sink” Approach To Remediation
My recommendations aren’t new. I’ve been saying this for a very long time. Don’t try to isolate one or two problems… Google is evaluating many factors when it comes to these broad core ranking updates. My advice is to surface all potential problems with your site and address them all. Don’t tackle just 20% of your problems. Tackle close to 100% of your problems. Google is on record explaining they want to see significant improvement in quality over the long-term in order for sites to see improvement. See my examples above for how that can work.

Here are some things you can do now (for improvement down the line). Again, this list should be familiar…

  • Review and address content quality. Surface all thin or low-quality content and handle appropriately. That could mean boosting content that’s low-quality, 404ing that content, or noindexing it.
  • Analyze your technical SEO setup and fix any problems you surface as quickly as you can. I’ve always said that what lies below the surface could be scary and dangerous SEO-wise. For example, canonical problems, render problems crawling problems, and more.
  • Objectively analyze your advertising situation. Do you have any aggressive, disruptive, or deceptive ads? Nuke those problems.
  • Are there user experience (UX) barriers on your site? Are you frustrating users? If so, remove all of the barriers and make your site easy to use. Don’t frustrate users. Remember, hell hath no fury like a user scorned.
  • On that note, are you meeting and/or exceeding user expectations? Review the queries leading to your site and then the landing pages receiving that traffic. Identify gaps and fill them.
  • Make sure experts are writing and reviewing your content (especially if you focus on YMYL topics). Google is clearly looking for expertise when it matches users with content.
  • Build the right links, and not just many links. It’s not about the quantity, it’s about quality. We learned that Google (partly) evaluates E-A-T via PageRank, which is from links across the web. Use a strong content strategy along with a strong social strategy to build links naturally. When you do, amazing things can happen.

Summary – The March 12 Update Was Huge. The Next Is Probably A Few Months Away
Google only rolled out three broad core ranking updates in 2018. Now we have our first of 2019 and it impacted many sites across the web. If you’ve been impacted by the March update, it’s important to go through the steps I listed and look to significantly improve your site over the long-term. That’s what Google wants to see (they are on record explaining this).

Don’t just cherry pick changes to implement. Instead, surface all potential problems across content, UX, advertising, technical SEO, reputation, and more, and address them as thoroughly as you can. That’s how you can see ranking changes down the line. Good luck.

GG

7 Examples of Untrackable Clicks From Google’s Ecosystem of Search, Chrome, and Feeds

By Leave a Comment

Information is powerful. And for SEOs, it can help inform, guide, and drive change. That’s why we’re always looking for more data from Google to better understand where traffic is coming from, what people are searching for, which elements in the search results are driving those clicks, and more. And that’s also why tools such as Google Search Console (GSC), Google Analytics, and a number of third-party tools are critically important for providing context to search traffic.

On that note, even though there are some outstanding third-party tools, we rely heavily on Google to provide that information. Third-party tools can be extremely helpful, but there’s nothing like getting your data directly from the belly of the beast.

Change Is Constant And So Are The SERPs, Chrome, and More
I recently wrote a post for my Search Engine Land column that demystified the metrics in Google Search Console (GSC). That included how Google calculates impressions, clicks, and position in the search results and how that translates to your reports. My column seemed to resonate with the SEO community, which supports the fact that we are all extremely interested (or maybe obsessed) about data from Google. It also underscores the point that we want to know exactly what we’re looking at in our reports.

So, data is one thing, but the correct interpretation of data is another. That leads me to the core point of this post – untrackable clicks from Google, often from new and interesting SERP features, Chrome, and feeds.

What Are Untrackable Clicks From Google?
Untrackable clicks are visits from SERP features, or Google’s ecosystem of Chrome and feeds, that don’t have specific reporting in Google Search Console (GSC) or Google Analytics. You can often see impressions and clicks for the query and page (if your content is ranking in Search), but the search feature isn’t trackable in Search Console (at least yet).

And for Google’s ecosystem beyond Search (like Chrome and Discover), they don’t have a logical place in GSC yet (and may never), and those visits can show up in Google analytics a number of ways (yet obviously don’t reveal which features they are coming from). It’s all a bit confusing…

Also, context is extremely important for SEOs and site owners when they are analyzing their organic search efforts. So, understanding when listings are ranking in certain SERP features, or within Google’s ecosystem, is important. Then you can better understand how users are getting to your content, analyze the content that’s ranking, and then craft strategies to potentially land more of those rankings.

Before we begin… the importance of posting feedback in GSC
Just to be clear, site owners and SEOs don’t have to sit around and twiddle their thumbs while untrackable clicks rack up in GSC. Collectively, we do have a voice. The Google Search Console team is a smart group of savvy Googlers that wants to provide the best functionality for site owners.

They have recently given us extremely powerful functionality in GSC like the index coverage reporting and the URL inspection tool, and I know they seek feedback from users about what they could add in the future.

So definitely post feedback in GSC when you see something wrong, if you have an idea for new functionality, etc. From what John Mueller has said many times, the team does receive and read your feedback and that feedback can help validate feedback they receive from other places. So, use the feedback form in GSC!

Submitting feedback in Google Search Console (GSC)

Below, I have provided a list of seven examples of untrackable clicks from Google (from the SERPs and from Google’s ecosystem of Chrome and Feeds). Note, I can’t cover every possible form of untrackable click in this post. The list will always be expanding based on innovations from Google… That said, I have provided seven of the top examples below.

Video Carousels and New “Video Packs”
The mobile SERPs are filled with carousels, which makes sense… Carousels provide a slick way to pack more listings into a single search result. One type of carousel that has taken off recently is the video carousel. They started booming in the SERPs in mid-2018 and are present for many queries now (some would argue they are present for too many queries, but that’s for another post).  

Also, I just noticed (and tweeted) that video packs are showing up in place of many carousels in the mobile search results (showing between 4 and 10 videos per block). That’s an interesting move and one you should definitely be aware of if video is important to your efforts.

For now, there’s no specific reporting for video carousels or new video packs in Search Console. The “video” filter in GSC under Search Type will only show you rankings from the video tab in the SERPs (video search), and not video carousels or packs from the core search results (10 blue links).

The video search type in GSC just filters video tab results.

From a video carousel standpoint (which is blended into the SERPs), if a listing shows up in a video carousel, and it’s scrolled into view, then it will register an impression. But again, your listing needs to scroll into view for that to happen. And there’s no filter for video carousels in GSC, so the clicks and impressions show up just like any other clicks and impressions from organic search.

It would be incredible to know when certain urls were ranking in video carousels and then obviously drill into the queries that yielded those rankings.  

From a tracking perspective, a video carousel filter would be very helpful. For example, in GSC’s performance reporting, you can currently filter AMP rich results and other rich results, but you can’t view video carousel data. I think based on the video carousel boom, a filter there would be extremely helpful. See a mockup below of what that could look like:

A video filter under Search Appearance could reveal video carousel or video pack data.

Discover Feed
Discover is a customized feed of news, information, articles, evergreen content, video, and more. It was previously called “the Google Feed” and you can find your Discover feed in the Google app and on the homepage of Google.com on your mobile device (when you are logged in). Google announced that 800M people now use Discover (as of September of 2018).

It’s important to note, that you don’t need to be in Google News to rank in Discover. Google is providing information based on your interests, sites you visited in the past, the topics you have selected to follow, entities you have selected to follow in the SERPs, and more. And it includes evergreen content that’s not new to the web, but might be new to you.

Discover feed in the Google search app.
Google providing more topics that users can follow
based on their interests.

Although many people are browsing Discover (some have even called it their new Instagram feed), and clicking through to articles from Discover, there’s currently no way to track that’s happening. It’s another form of untrackable click from across Google’s ecosystem.

For example, I’ve seen articles from various SEOs I know in my Discover feed, I’ve clicked through to those articles, but they unfortunately have no idea that happened. In Google Analytics, visits from Discover can show up as direct traffic, or they can be attributed to your first visit to the site (which could be organic search). So GA doesn’t really help here.

Understanding traffic in Discover would help SEOs understand when Google has decided to include their content in someone’s feed, how much traffic (or return traffic) they are getting via Discover feeds, and more. It’s not organic search traffic, but could be based on iniitial visits from organic search.

Therefore, some type of tracking in GSC would be incredible. Maybe there could be another report for Discover. Remember, over 800M people use Discover on a regular basis now… Here’s a quick mockup of a new section for Discover data:

Discover tracking in GSC would be an interesting addition.

Discover Update: 2/16/19
After publishing this post, a reader posted a comment that he found a blank Discover report *in GSC now*. He pointed me to this tweet from JR Oakes as reference. So Google is actually testing performance reporting for Discover! There’s no data yet, but that’s still incredible news.

Note, Google has tested features like this before in GSC and never rolled them out. So, it’s possible this might not see the light of day. Time will tell. But needless to say, I was pretty excited to see it. And hey, my mockup above wasn’t that far-fetched! :)

Discover reporting is being tested in GSC now! The reporting is blank for most sites,
but it’s great to see this being tested.

Interesting finds
I call this the search feature nobody is talking about… yet it’s been popping up more and more recently. When searching Google, you will often see “interesting finds” in the mobile search results as a module containing either three or four articles (with thumbnails). Note, I just noticed three listings showing up versus four and will share more about that on Twitter soon. And if you are using AMP, those AMP urls will rank in the “interesting finds” SERP feature. Notice the AMP icon below for my listings.

A four pack of “interesting finds” (with AMP urls).
Interesting finds with three listings versus four.

“Interesting finds” can also contain a link to ten more articles, which takes you to a Discover-like feed for the topic. It can really grab a user’s attention. That’s where you can see up to ten articles on the topic. And tying this to Discover, you can also follow the topic! Remember when I called this Google’s ecosystem?

10 more articles in “interesting finds” along with
a follow button (for Discover).

The problem is that site owners have no idea users are coming from that SERP feature. The impression and click will show up just like any other listing in the 10 blue links. The position will be the position of the “interesting finds” block. And in Google Analytics, the visit will look like any other visit from Google organic.

From a tracking perspective, it would be optimal to know that your articles are ranking in “interesting finds” for a number of reasons. For example, since Google thinks enough of the content to surface it in the “interesting finds” module for a specific topic, it would be great to analyze that content to help refine your content strategy (and to possibly gain more “interesting finds”).

In addition, it would be great to know when an article drops out of “interesting finds”, which can yield a drop in impressions and clicks. If you can’t track that, then site owners and SEOs could think something else happened that cause the drop in traffic. And that can lead to a lot of confusion.

“More like this” or “Related Pages” in the Google Search App
I’ve been meaning to write a post dedicated to this feature, since I know many don’t realize that it can be driving traffic to their sites. When you’re in the Google search app and you visit a page, you can always click the “More like this” icon in the menu bar. The feature was moved to the top of the app last year, so it’s *sometimes* in a prominent location.

The reason I say sometimes is because I just noticed today that it moved again! Now it’s available when you scroll down a page (you’ll see a “View 10 related pages”, which you can pull up from the bottom of the viewport). See below:

Related Pages feature in the Google App
(when available at the bottom of an article)
Related Pages feature when expanded
(pulled up from the bottom of the viewport).
“More like this” in the Google app
(via an icon at the top of the page)

When clicking that icon, or pulling up the related pages bar, you will see a list of links that Google believes are extremely relevant to the content at hand. It’s basically a form of “related articles”, but from Google and not third-party services. Similar to other features I’m covering in my post, you will never know that users originated from “More like this” listings. You will just see a standard visit in Google Analytics, and since this is from Chrome and not Search, you won’t see impressions or clicks at all in Google Search Console.

It would be great to know how much traffic is coming from this feature and possibly the theme or topic that yielded a “More like this” ranking. And again, it would be great to know when articles dropped out of this feature for various topics, so site owners can be informed about a drop in traffic possibly originating from the “More like this” or “Related Pages” feature in Chrome.  

Featured snippets, facets in featured snippets, and more
Although I was hopeful that Google would eventually roll out native featured snippet tracking in Google Search Console (GSC), that never happened. And with featured snippets taking up large amounts of real estate, and driving a lot of traffic, it’s hard to believe that site owners must rely on third-party tools to know when their content ranks in a featured snippet.

Side note, I recently wrote a post about how to use SEMrush to surface more of your featured snippets than what’s being reported by default in third-party tools. It’s a strong way to go, but let’s face it, having data directly from Google would be optimal.

Beyond just knowing if you are ranking in a featured snippet, it would be great to know the type of featured snippet. Featured snippets come in many forms, so understanding paragraph versus bullets versus tables versus quasi-knowledge panels would be amazing.  

Now, Google’s featured snippet algorithm is extremely temperamental, and featured snippets can change quickly (and often), but it would still be good to have some information from Google about this.

Google is also showing facets in some featured snippets, which trigger additional featured snippets. Try and say that ten times fast. :) There’s obviously no reporting right now for that either. And that can sometimes include video facets too, which trigger suggested clips that play in a lightbox above the SERPs.

Featured snippets facets in action.
When clicking a facet, a new featured snippet is triggered.

All of this would be great for site owners and SEOs to understand to help guide their content efforts. For example, which posts are yielding featured snippets, what types of content work best, which format is best, is a post ranking as a facet versus the default featured snippet, and more about the featured snippet sequence (e.g. are posts moving from the 10 blue links to a facet and then to a featured snippet?)

People Also Ask (PAA)
When searching on Google, you will often see People Also Ask modules, which contain a second level of search listings (presented in a similar format to a featured snippet). When expanding a query in PAA, you will see more information, a link to the destination site, and then an option to search Google for that query. But just like featured snippets, you will have no idea that your content is ranking in PAA modules (not from Google anyway).

People Also Ask cannot be tracked in GSC.
PPA triggers a new listing in a similar format to featured snippets.

If you read my post on SEL about how Google calculates impressions, clicks, and position, then you learned that each listing in a block element will take on the position of the block. And I also explained how listings need to be revealed in order to gain impressions. So theoretically, PAA can actually make your stats look a little funny.

You might show up ranking #2 in PAA, or you might not, since the user didn’t expand your PAA listing. It’s a great reason why adding some type of tracking in GSC for PAA would be helpful. And remember, there’s a form of infinite scroll with PAA. That means your listing could have a position of 2, but it might be the 15th PAA listing. Strange, but true.

Local Pack Listings (GMB)
UTM tracking is getting killed off via the recent change announced for GSC (more on that soon). So local listings will also be harder to track via GSC now. This has been a hot topic recently, since urls with utm parameters have seen some strange reporting in GSC over the past several weeks. Impressions dropped off a cliff, while clicks remained stable. For example, notice the drop in impressions below:

The reporting issue now makes more sense as Google announced it’s consolidating metrics in GSC to canonical urls. So for this situation, the utm urls are being canonicalized to the core urls (or will in the near-future), so their metrics will shift to the canonical urls. It’s not working perfectly yet, but that’s what is going to happen.

So, local SEOs will have no idea when their urls are showing up in the local pack, how much traffic they are driving, etc. One solution that’s been brought up on Twitter was the idea of adding a local filter to the search appearance functionality in GSC within the performance reporting. John Mueller actually liked that idea.

To me, that’s a great solution for the local issue, and for other SERP features I’ve mentioned in this post. And just like I explained earlier, definitely go and submit feedback in GSC if that’s something you would like to see. The GSC product team receives that feedback and it can help validate feedback they are receiving from other sources.

Third-party tools as a stop-gap:
Based on what I’ve explained above, I hope Google can provide some of this data in GSC. Again, I think site owners and SEOs would greatly value understanding where clicks are coming from in order to fine-tune their digital marketing efforts.

Thankfully, we have some outstanding third-party tools in the SEO industry that can at least identify when your site is ranking within certain SERP features. I won’t go too deep here, but I did want to provide a short list containing some of my favorites below. Note, you obviously can’t see traffic leading to your site via these tools, but they can help you understand when your listings are part of certain SERP features.

Which third-party tools track SERP features?
My favorite tools that enable site owners and SEOs to filter rankings by SERP features include:

  • SEMrush (I’ve often called SEMrush the swiss army knife of SEO tools)
  • Ahrefs
  • Sistrix
  • Then there are a number of dedicated rank-tracking tools that can help you surface this information as well. For example, RankRanger and STAT (now part of Moz) can surface a number of SERP features based on keywords you are manually tracking.

Again, you can’t see every query or SERP feature via third-party tools, but they can help you understand more of what’s going on in Google Land.

Summary – Seeking clarity from SERP features and Google’s ecosystem of Chrome and Feeds
As I explained in this post, context is very important. And understanding which features are driving traffic can help provide a clearer picture of traffic from Google Search, Chrome, Discover, and more. I hope this feedback reaches the GSC team and they decide to expand which features are tracked in GSC. That would be incredible and could help many site owners and SEOs better track their efforts. I’m hoping for the best!

GG


A Holly Jolly Load Balancer Christmas – How Google Treated A Major Site Performance Problem From A Crawling And Ranking Perspective [SEO Case Study]

By Leave a Comment

Ah, the holidays. Who doesn’t love a festive atmosphere, ugly Christmas sweaters, eggnog, singing Christmas carols, and major performance problems that can take down your site?

Wait, what was that last part??

The holidays are always interesting for me SEO-wise. Every few years, I receive an SOS from a site owner right around Christmas Day based on a weird drop in traffic or rankings from Google. This year, it was on December 27 from a company I helped several years ago. It’s a large-scale site that has seen a lot of volatility historically, but has done a good job at turning things around. They have battled major algorithm updates in the past, and now battle new SERP features from Google itself.

Just two days after Christmas, I received a message from the site owner asking if there was some type of update over the holidays. Traffic dipped sharply on Christmas Day, which it typically wouldn’t do based on their niche, and had gone down more and more since. Traffic typically only goes down by about 20% on Christmas Day and it was down 50% this year. That’s typically not the case from a historical perspective for the site and they were worried that an algorithm update impacted the site again.

So I rushed to check their stats, noticed the drop in traffic, and quickly started checking the sources of traffic that dropped. The first thing I noticed was that the drop wasn’t just from Google… it was from all sources. That’s actually a good thing and we could quickly rule out some type of Google algorithm update.

Then I started checking the top landing pages that dropped and it wasn’t long before I noticed a serious performance problem. Some of the pages simply wouldn’t load. Then I searched Google for queries leading to the pages and tried to click through from the SERPs. I experienced the same issue… the site was simply hourglassing.

I messaged the site owner and explained the major performance problems I was experiencing and that the drop was from all traffic sources, and not just from Google. He took that information to his CTO so they could dig in and start isolating the problem. It wasn’t long before I heard back.

A Holly Jolly Load-Balancer Christmas!
No, that’s not the name of the next Christmas Vacation movie, although it does have a nice ring to it. :) The site owner and CTO uncovered a load balancer issue that was causing the performance problems we were seeing on the site. And that was causing a big drop in traffic as visitors couldn’t load many of the pages. And again, this is a site that people visit during the holidays, and on Christmas day, based on the type of content the site provides.

So, the good news is that it wasn’t some type of evil elf algorithm update. But the bad news was that the load balancer issue was causing many pages not to load for users and Googlebot. My client went to work on resolving the load balancer issue, but they couldn’t help but wonder how Google would treat the site from a rankings perspective.

For example, when Google sees a major performance problem over time, and it knows users cannot load the pages, some important questions come to mind. Will Google drop the site’s rankings, will the site plummet in the short-term, only to come back when the performance problems are corrected, will there be long-term damage SEO-wise, and how will crawling and indexing be impacted?

We’ve heard Google’s John Mueller explain situations related to this in the past. For example, John explained that Google can slow crawling when it comes across many 5XX errors. Here’s a tweet from John about this:

And that goes for 503s as well (a response code sites can return when maintenance is being completed and the site is down). John has explained that if Google sees 503s for an extended period of time, it can start dropping urls from the index. Note, returning 503s for a short period of time is totally fine (and is a good approach to use when you know your site will be down for maintenance). Just make sure you’re not returning 503s for an extended period of time.

Since this was a load balancer issue, and some visitors were able to load pages correctly, my client didn’t take the site offline (and didn’t use 503s). But it’s important to understand how Google would treat a site being down over an extended period of time. Here’s a video where John explains this (at 37:28 in the video):

And here’s John explaining how Google can automatically slow crawling when it sees that urls are temporarily unavailable. He also explains that Google will stop crawling if it sees that happening for a robots.txt file. Then Google will automatically increase crawling when the urls return 200s again. Here’s the video of John explaining this (at 38:43 in the video):

So what would happen in this case? Needless to say, we were eager to find out.

Tracking The SEO Impact Of Major Performance Problems
As you can probably guess, I was ultra-interested in seeing how this played out SEO-wise. So, I began tracking the situation on several levels to see how the load balancer issue was impacting the site from an SEO standpoint. For example, would rankings drop, would crawling slow down, and how would this all look when the performance problems were fixed?

Below, I’ll take you through the timeline, with data and screenshots, to show you how Google treated the problem. Note, every site is different, so this may not apply to all performance problems you run into. But, it should help you understand how those performance problems could impact crawling, indexing, and ranking. It’s also worth noting that the duration of performance problems would surely impact this situation. For this case, the site owner and CTO completely fixed the problem within seven days. But, it wasn’t all fixed at one time. More on that soon. Last, I unfortunately don’t have logs from the site yet. I’m still working on getting them, though, and will update this post if I do.

Blue Christmas – The Timeline of Events (with data and screenshots)
First, here is the drop in traffic when comparing year over year. The site usually doesn’t drop very much during Christmas Day (or week). You can see a seasonal drop of about 20% when compared to normal weeks, where this year yielded a drop of 50%. Yep, something was up…

YOY trending shows the unusual drop on Christmas Day in 2018.

Crawl Stats in GSC:
Here was the spike in time downloading a page and the drop in pages crawled per day. Pretty amazing to see that graph, right?

Crawl stats in GSC show a spike in time spent downloading a page
while pages crawled per day drop off a cliff.

And here were the impressions and clicks in GSC (which remained strong). This supports the notion that rankings were not being impacted during the performance problem. Impressions would have dropped if that were the case. I was also checking rankings manually and saw rankings remain strong overall. It seems that Google was pretty cool with understanding there was a performance problem and didn’t quickly impact rankings (even when it knew some users were running into page loading problems).

GSC Performance reporting reveals stable impressions and clicks from Google
while the performance problem was in place.

And here was search visibility trending in SEMrush over the timeframe, which supports the fact that rankings were not impacted in the short-term. Search visibility was stable throughout the performance problem:

Search visibility trending in SEMrush is stable throughout the problem (no drop in rankings).

And here is Sistrix trending, which is also stable throughout the problem:

Search visibility trending in Sistrix is also stable throughout the problem.

Note, the load balancer problem was not fixed all at one time. The situation improved over a five to seven-day period. So here are the crawl stats as the issue started to get resolved. Notice crawling increases, but not back to normal. That’s because the performance problems were still there to some extent. Notice time downloading a page improves, but also not back to normal.

Pages crawled per day increase as time spent downloading a page decreases
(as the load balancer problem is being fixed).
Pages crawled per day and time spent downloading a page continue
to improve as the performance problem is fixed.

And then once the problem was completely fixed, notice that the crawl stats return to normal. Time downloading a page drops back down to normal levels and pages crawled per day rise to normal levels. Awesome, all is good again in Google Land.


Pages crawled per day and time spent downloading a page return to normal levels
when the load balancer problem was completely fixed.

And here is what Google organic traffic looked like from before the load balancer issue to after it was resolved. Notice the drop during Christmas, a further drop, and then traffic returns to normal after the problem was fixed. There was no impact to Google organic traffic, rankings, etc. based on the performance problem. That was great to see.

Traffic from Google organic returns to normal once the
load balancer problem was completely fixed.

Key learnings… and Spiked Eggnog:
Needless to say, it was fascinating to watch how Google reacted to the major performance problems over time. Here are some key learnings from the situation:

  • Google understands that temporary glitches can happen (like the performance problems this site was experiencing).
  • Google can slow crawling, when needed, and check back to see when it can crawl more. The crawl stats in GSC clearly showed that happening.
  • For this situation (only a week or so), the site wasn’t impacted SEO-wise. Rankings remained strong, search visibility was stable, and as the problem was fixed, Google returned crawling back to normal levels.
  • Google organic traffic then returned to normal levels as the performance problem was fixed completely. There was no impact rankings-wise based on this incident.
  • That said, if this problem remained beyond a week or two, it’s hard to say how rankings would be impacted. Google could very well begin to drop rankings as it wouldn’t want to send users to a site or pages that don’t resolve. But, if you have a short-term performance issue, it seems you should be ok. I would just work to fix the problems as quickly as you can.
  • This is a reminder that Murphy’s Law for SEO is real. The load balancer problem happened on Christmas Day. You just need to be prepared for Murphy to pay a visit. And then move quickly to rectify the problems he brings along.

Summary – All I want for Christmas is to never deal with a load balancer issue again.
If you run into serious performance problems and are worried about the SEO-impact, know that Google does understand that bad things happen from time to time. As Google’s John Mueller explained, Google can slow crawling while the problems exist and then check back to see when it can return crawling to normal.

For this specific case, that’s exactly what we saw. Rankings and search visibility remained strong, but crawling slowed. Then as the performance problems were fixed, crawling returned back to normal. And all was good again in Google Land.

So, until next holiday season… here’s ho-ho-hoping you don’t run into this type of problem anytime soon. I know everyone involved with this case feels the same way. :)

GG

Searching For Buried Treasure – How To Find More Of Your Featured Snippets Using Google Search Console (GSC), Analytics Edge, And SEMrush Position Tracking

By Leave a Comment

Searching for buried SEO treasure.

A few years ago, I wrote a post about how to surface your featured snippets via SEMrush’s powerful SERP features widget. I love using SEMrush to plug in any domain and view (many of) the featured snippets it has. It’s a quick an easy way to conduct competitive analysis from a featured snippets standpoint. Remember, GSC does not provide a way to filter for featured snippets. Therefore, you must rely on third party tools to surface them for you.

Now, SEMrush’s SERP features widget is awesome, but here’s the issue. SEMrush, and other tools, don’t surface ALL of your featured snippets automatically. And that’s especially true for smaller to medium-sized sites. Unfortunately, many site owners are left in the dark with how many featured snippets they have and which queries and landing pages are yielding featured snippets.

SERP features widget in SEMrush.

It’s important to understand the featured snippets you have attained so you can analyze the content, the snippets, and try to gain more. As I’ve covered in various posts, as well as in my SMX presentations, featured snippets can drive a boatload of traffic and gain near-instant credibility in the SERPs. Position 0 is powerful.

So, with GSC not providing a way to filter by featured snippets and competitive analysis tools not surfacing all of a site’s featured snippets automatically, what’s a site owner to do?

The answer is to USE THE TOOLS TOGETHER TO UNCOVER MORE OF YOUR FEATURED SNIPPETS!

That’s what I’ll cover below. It’s quick, easy, and you can continually use this process to hunt down your latest featured snippets. Let’s begin.

How To Use GSC, Analytics Edge, And SEMrush Position Tracking To Identify Featured Snippets
SEMrush provides a ton of functionality for SEOs and site owners. For our purposes today, we’ll be using its Position Tracking functionality based on GSC data to surface more of your wonderful featured snippets. The process involves:

  • Exporting your top queries from Google Search Console for the past 28 days.
  • Filtering them by average position and click through rate.
  • Setting up position tracking for your domain in SEMrush and using the filtered queries from GSC as your keyword list.
  • Letting SEMrush position tracking do its thing!
  • Reviewing the reporting in SEMrush, which will surface any queries yielding featured snippets (along with the landing pages that are ranking for them).

Exporting Google Search Console Data
Analyzing your queries in Google Search Console is obviously extremely important. The problem is that you are limited to the top one thousand results per report. That’s severely limiting for most sites. In order to get around that limit, you can tap into the Search Console API to download ALL of your queries. There are several ways to accomplish this, but my favorite is using Analytics Edge in Excel. It’s a cost-effective solution that works incredibly well (and fast).

I’ve written several blog posts about how to use Analytics Edge, so go and check those posts out to learn how to download all of your queries. Once you have Analytics Edge installed, it should only take a minute or two to download all of your queries. You’ll end up having a spreadsheet full of glorious GSC keyword data. For example, I exported 66K queries in about thirty seconds. :)

Exporting many queries from GSC via Analytics Edge.

Next, we want to gather our list of queries to use in SEMrush for position tracking. Featured snippets in GSC will be listed in position 1. Google will provide the ranking of the actual featured snippet here and NOT the organic listing. For example, you might have the featured snippet, but your organic listing might be #4. GSC would provide #1 for the query. You can read more about how Google determines clicks and impressions in this help center document to learn how GSC provides the position for featured snippets.

Therefore, you should first filter your spreadsheet by average position and use a number filter selecting “is less than 2”. This will give us any queries where you site is ranking between 1 and 2 over the past 28 days.

Filtering by average position in Excel.

Next, layer on a second filter in your spreadsheet for click through rate. Featured snippets typically have a higher click through rate since they are given special SERP treatment and take up a lot of SERP real estate. So for the second filter, use “greater than 20%” for click through rate.

Filtering by click through rate in Excel.

Quick Tip:
You can also filter out branded queries if you want for this exercise. Many branded queries will rank highly and also have a high click through rate, but won’t yield featured snippets. If you want to save some space in SEMrush for other queries (since there is a limit for position tracking based on your package), then feel free to filter them out. You can do that by adding a “does not contain” text filter on your queries column and enter your brand.

Filtering out brand queries in Excel.

Now that you have a list of queries ranking highly (position 2 or less) that also have a high click through rate, then you are ready to hunt down featured snippets and uncover buried treasure in SEMrush. Let’s track some positions.

How To Set Up Position Tracking In SEMrush (And Find Featured Snippets)
First, you’ll want to create a project for the domain you are tracking. You can find the Projects link in the left-side navigation in SEMrush. Once you create a project, you’ll be presented with all of the different tools you can use in your project. We’re going to be focused on Position Tracking so we can identify which queries are yielding featured snippets.

Projects in SEMrush.

Next, click Position Tracking and go through the process of adding your filtered keyword list. You can start by just selecting desktop tracking in the United States, or tailor that for your own needs. I typically add desktop tracking first, and then layer on mobile. It’s very easy to do since you can just use the desktop list by clicking a button in SEMrush when setting up mobile.

Setting up position tracking in SEMrush.

Adding keywords when setting up position tracking in SEMrush.

Quick Tip:
Depending on your package in SEMrush, there’s a limit for the number of keywords you can track. For example, a Pro account can track 500 keywords, a Guru account can track 1500 keywords, etc. For this specific project, you won’t be focused on long-term tracking, so you can always swap out keywords to simply check if they are yielding featured snippets. So just make sure your initial keyword list is under the limit you have in your package. You can always swap out one list for another to uncover more featured snippets.

The Easy Part – Let SEMrush Position Tracking Do Its Thing!
Once you enter your keywords and launch position tracking, you only have a to wait a few minutes for your featured snippet data to arrive (if you have any!) Once SEMrush tracks the keywords in your list, you’ll have a boatload of data for those queries, including if they are yielding featured snippets.

There are multiple ways to find your current featured snippets in the reporting based on the keyword list you just entered. From the Overview screen, you can view the entire list of keywords along with their position, the SERP features for that query, and if your site has any of those important SERP features (like featured snippets).

For example, you’ll see a crown icon if the SERP contains a featured snippet for the query at hand. And, you’ll see a second crown next to your ranking if you have won the featured snippet.  You will also see the landing page that’s yielding the featured snippet.

Finding featured snippets in SEMrush position tracking.

You can also click the featured snippet tab to see a full breakdown of your featured snippets reporting. That will include opportunities you have to gain featured snippets from the competition, as well as when your site is already featured. Clicking the button labeled “Already featured” will filter just the keywords where your site has the featured snippet.

Viewing the featured snippets tab in SEMrush.

And clicking the position and crown enables you view a snapshot of the SERP captured by SEMrush. Awesome, right?

Viewing the SERP snapshot for featured snippets in SEMrush.

Summary – Finding featured snippet buried treasure via GSC and SEMrush
SEMrush and other tools can automatically surface some of your featured snippets, which is great. But to gain a stronger view of your featured snippet coverage, it’s smart to combine Google Search Console data with SEMrush position tracking. You might be surprised when viewing all of the featured snippets you have right now. Now go dig up that buried treasure. :)

GG

 

How To Find The True Size Of Your Site Using GSC’s Index Coverage Reporting (And Why It’s Important For SEO)

By Leave a Comment

How large is your site SEO-wise?

So, how large is your site? No, how large is it really??

When speaking with companies about SEO, it’s not long before I ask that important question. And I often get some confused responses based on the “really” tag at the end. That’s because many site owners go by how many pages are indexed. i.e. We have 20K, 100K, or 1M pages indexed. Well, that’s obviously important, but what lies below the surface is also important. For example, Googlebot needs to crawl and process all of your crawlable urls, and not just what’s indexed.

Site owners can make it easy, or very hard, for Google to crawl, process, and index their pages based on a number of factors. For example, if you have one thousand pages indexed, but Google needs to crawl and process 600K pages, then you might have issues to smooth over from a technical SEO perspective. And when I say “process”, I mean understand what to do with your urls after crawling. For example, Google needs to process the meta robots tag, rel canonical, understand duplicate content, soft 404s, and more.

By the way, I’ll cover the reason I said you might need to smooth things over later in this post. There are times a site has many urls that need to be processed that are being properly handled. For example, noindexing many pages that are fine for users, but shouldn’t be indexed. That could be totally ok based on what the site is trying to accomplish. But on the flipside, there may be sites using a ton of parameters, session IDs, dynamically changing urls, or redirects that can cause all sorts of issues (like creating infinite spaces). Again, I’ll cover more about this soon.

And you definitely want to make Google’s job easier, and not harder. Google’s John Mueller has mentioned this a number of times over the past several years. Actually, here’s a video of John explaining this from last week’s webmaster hangout video. John explained that you don’t want to make Google’s job harder by having it churn through many urls. Google needs to crawl, and then process, all of those urls, even when rel canonical is used properly (since Google needs to first crawl each page to see rel canonical).

Here’s the clip from John (at 4:16 in the video):

John has also explained that you should make sure Google can focus on your highest quality content versus churning through many lower quality urls. That’s why it’s important to fully understand what’s indexed on your site. Google takes all pages into account when evaluating quality, so if you find a lot of urls that shouldn’t be indexed, take action on them.

Here’s another video from John about that (at 14:54 in the video):

Crawl Budget – Most don’t need to worry about this, but…
In addition, crawl budget might also be a consideration based on the true size of your site. For example, imagine you have 73K pages indexed and believe you don’t need to worry about crawl budget too much. Remember, only very large sites with millions of pages need to worry about crawl budget. But what if your 73K page site actually contains 29.5M pages that need to be crawled and processed? If that’s the case, then you actually do need to worry about crawl budget. This is just another reason to understand the true size of your site. See the screenshot below.

Potential crawl budget issues.

How Do You Find The True Size Of Your Site?
One of the easiest ways to determine the true size of your site is to use Google’s index coverage reporting, including the incredibly important Excluded category. Note, GSC’s index coverage reporting is by property… so make sure you check each property for your site (www, non-www, http, https, etc.)

Google’s Index Coverage
I wrote a post recently about juicing up your index coverage reporting using subdirectories where I explained the power of GSC’s new reporting. It replaces the index status report in the old GSC and contains extremely actionable data. There are multiple categories for each property in GSC in the reporting, including:

  1. Errors
  2. Valid with warnings
  3. Valid and indexed
  4. Excluded

By reviewing ALL of the categories, you can get a stronger feel for true size of your site. That true size might make total sense to you, but there are other times where the true size might scare you to death (and leave you scratching your head about why there are so many urls Google is crawling).

Below, I’ll run through some quick examples of what you can find in the index coverage reporting that could be increasing the size of your site from a crawling and processing standpoint. Note, there are many reasons you could be forcing Google to crawl and process many more pages than it should and I’ll provide just some quick examples in this post. I highly recommend going through your own reporting extensively to better understand your own situation.

Remember, you should make it easier on Googlebot, not harder. And you definitely want to have Google focus on your most important pages versus churning through loads of unimportant urls.

Errors
There are a number of errors that can show up in the reporting, including server errors (500s), redirect errors, urls submitted in sitemaps that are soft 404s, have crawl issues, are being blocked by robots.txt, and more. Since this post is about increasing the crawlable size of your site, I would definitely watch out for situations where Google is surfacing and crawling urls that should never be crawled (or urls that should resolve with 200s, but don’t for some reason).

There are also several reports in this category that flag urls being submitted in xml sitemaps that don’t resolve correctly. Definitely dig in there to see what’s going on.

Errors in GSC's index coverage reporting.

Indexed and Valid
Although this report shows all urls that are properly indexed, you definitely want to dig into the report titled, “Indexed, not submitted in sitemap”. This report contains all urls that Google has indexed that weren’t submitted in xml sitemaps (so Google is coming across the urls via standard crawling without being supplied a list of urls). I’ve found this report can yield some very interesting findings.

For example, if you have 200K pages indexed, but only 4K are submitted in xml sitemaps, then what are the other 196K urls? How is Google discovering them? Are they canonical urls that you know about, and if so, should they be submitted in sitemaps? Or are they urls you didn’t even know existed, that contain low-quality content, or autogenerated content?

I’ve surfaced some nasty situations by checking this report. For example, hundreds of thousands of pages (or more) indexed that the site owner didn’t even know were being published on the site. Note, you can also fine some urls there that are fine, like pagination, but I would definitely analyze the reporting to gain a deeper understanding of all the valid and indexed urls that Google has found. Again, you should know all pages indexed on your site, especially ones that you haven’t added to xml sitemaps.

Indexed not submitted in GSC's index coverage reporting.

Excluded – The Mother Lode
I’ve mentioned the power of the Excluded category in GSC’s index coverage reporting before, and I’m not kidding. This category contains all of the urls that Google has crawled, but decided to exclude from indexing. You can often find glaring problems when digging into the various reports in this category.

I can’t cover everything you can find there, but I’ll provide a few examples of situations that could force Google to crawl and process many more urls than it should. And like John Mueller said in the videos I provided above, you should try and make it easy for Google, focus on your highest quality content, etc.

Infinite spaces
When I first fire up the index coverage reporting, I usually quickly check the total number of excluded urls. And there are times that number blows me away (especially knowing how large a site is supposed to be).

When checking the reporting, you definitely want to make sure there’s not an infinite spaces problem, which can cause an unlimited number of urls to be crawled by Google. I wrote a case study about an infinite spaces problem that was impacting up to ten million urls on a site. Yes, ten million.

Infinite spaces based on site search problems.

Sometimes a site has functionality that can create an unlimited number of urls with content that’s not unique or doesn’t provide any value. And when that happens, Google can crawl forever… By analyzing the Excluded category, you can potentially find problems like this in the various reports.

For example, you might see many urls being excluded in a specific section of the site that contain parameters used by certain functionality. The classic example is a calendar script, which if left in its raw form (and unblocked), can cause Google to crawl infinitely. You also might find new urls being created on the fly based on users conducting on-site searches (that was the problem I surfaced in my case study about infinite spaces). There are many ways infinite spaces can be created and you want to make sure that’s not a problem on your site.

Parameters, parameters, parameters
Some sites employ a large number of url parameters, when those parameters might not be needed at all. And to make the situation worse, some sites might not be using rel canonical properly to canonicalize the urls with parameters to the correct urls (which might be the urls without parameters). And as John explained in the video from earlier, even if you are using rel canonical properly, it’s not optimal to force Google to crawl and process many unnecessary urls.

Or worse, maybe each of those urls with parameters (that are producing duplicate content) mistakenly contain self-referencing canonical tags. Then you are forcing Google to figure out that the urls are duplicates and then handle those urls properly. John explained that can take a long time when many urls are involved. Again, review your own reporting. You never know what you’re going to find.

When analyzing the Excluded reporting, you can find urls that fit this situation in a number of reports. For example, “Duplicate without user-selected canonical”, “Google chose a different canonical than user”, and others.

Parameters in GSC's index coverage reporting.

Excessive redirects
There are times you might find a massive number of redirects that you weren’t even aware were on the site. If you see a large number of redirects, then analyze those urls to see where they are being triggered on the site, why the redirects are there, and determine if they should be there at all. I wouldn’t force both users and Googlebot through an excessive number of redirects, if possible. Make it easy for Google, not harder.

Redirects in GSC's index coverage reporting.

And some of those redirects might look like this… a chain of never-ending redirects (and the page never resolves). So one redirect might actually be 20 or more before the page finally fails…

Redirects chain for url that never resolves.

Malformed URLs
There have been some audits where I find an excessive number of malformed urls when reviewing the Excluded reporting. For example, a coding glitch that combines urls by accident, leaves out parts of the url, or worse. And on a large-scale site, that can cause many, many malformed urls to be crawled by Google.

Here is an example of a malformed url pattern I surfaced in a recent audit:
https://www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/www.domain.com/blog/category/some-blog-post-title/

Soft 404s
You might find many soft 404s on a site, which are urls that return 200s, but Google sees them as 404s. There are a number of reasons this could be happening, including having pages that should contain products or listings, but simply return “No products found”, or “No listings found”. These are thin pages without any valuable content for users, and Google is basically seeing them as 404s (Page Not Found).

Note, soft 404s are treated as hard 404s, but why have Google crawl many of them when it doesn’t need to? And if this is due to some glitch, then you could continually force Google to crawl many of these urls.

Example of a soft 404.

Like I said, The Mother Lode…
I can keep going here, but this post would be huge. Again, there are many examples of problems you can find by analyzing the Excluded category in the new index coverage reporting. These were just a few examples I have come across when helping clients. The main point I’m trying to make is that you should heavily analyze your reporting to see the true size of your site. And if you find problems that are causing many more urls to be published than should be, then you should root out those problems and make crawling and processing easier for Google.

Make it easy for Google.

Sometimes Excluded can be fine: What you don’t need to worry about…
Not everything listed in the index coverage reporting is bad. There are several categories that are fine as long as you expect that behavior. For example, you might be noindexing many urls across the site since they are fine for users traversing the site, but you don’t want them indexed. That’s totally fine.

Or, you might be blocking 60K pages via robots.txt since there’s no reason for Googlebot to crawl them. That’s totally fine as well.

And how about 404s? There are some site owners that believe that 404s can hurt them SEO-wise. That’s not true. 404s (Page Not Found) are totally normal to have on a site. And for larger-scale sites, you might have tens of thousands, hundreds of thousands, or even a million plus 404s on your site. If those urls should 404, then that’s fine.

Here’s an example of a site I helped with many 404s at any given time. They do extremely well SEO-wise, but based on the nature of their site, they 404 a lot of urls over time. Again, this is totally fine if the urls should 404:

Massive number of 404s on site doing well in organic search.

There have been several large-scale clients I’ve helped that receive over a million clicks per day from Google that have hundreds of thousands of 404s at any given time. Actually, a few of those clients sometimes have over one million 404s showing in GSC at any given time. And they are doing extremely well SEO-wise. Therefore, don’t worry about 404s if those pages should 404. Google’s John Mueller has covered this a number of times in the past.

Here’s a video of John explaining this (at 39:50 in the video):

There are other categories that might be fine too, based on how your own site works. So, dig into the reporting, identify anything out of the ordinary, and then analyze those situations to ensure that you’re making Google’s life easier, and not harder.

Index Coverage Limits And Running Crawls To Gain More Data
After reading this post, you might be excited to jump into GSC’s index coverage reporting to uncover the true size of your site. That’s great, but there’s something you need to know. Unfortunately, there’s a maximum one thousand rows per report that can be exported. So, you will be severely limited with the data you can export once you drill into a problem. Sure, you can (and should) identify patterns of issues across your site so you can tackle those problems at scale. But it still would be amazing to export all of the urls per report.

I know Google has mentioned the possibility of providing API access to the index coverage reporting, which would be amazing. Then you could export all of your data, not matter how many rows. In the short-term, you can read my post about how to add subdirectories to GSC in order to get more data in your reporting. It works well and you can read that post to learn more.

And beyond that, you can use any of the top crawling tools to launch surgical crawls into problematic areas. Or, you could just launch enterprise crawls that tackle most of the site in question. Then you can filter by problematic url type, parameter, etc. to surface more urls per category.

As I’ve mentioned many times in my posts, my three favorite crawling tools are:

  • DeepCrawl (where I’m on the customer advisory board). I’ve been using DeepCrawl for a very long time and it’s especially powerful for larger-scale crawls.
  • Screaming Frog – Another powerful tool that many in the industry use for crawling sites. Dan and his crew of amphibious SEOs do an excellent job at providing a boatload of functionality in a local crawler. I often use both DeepCrawl and Screaming Frog together. 1+1=3
  • Sitebulb – The newest kid on the block across the three, but Patrick and Gareth have created something special with Sitebulb. It’s an excellent local crawling tool that some have described as beautiful mix between DeepCrawl and Screaming Frog. You should definitely check it out. I use all three extensively.

 Summary – The importance of understanding the true size of your site.
The next time someone asks you how large your site is, try to avoid giving the quick response with simply pages indexed. If you’ve gone through the process I’ve documented in this post, the number might be much larger.

It’s definitely a nuanced answer, since Google might be churning through many additional urls based on your site structure, coding glitches, or other SEO problems. But you should have a solid grasp on what’s going on and how to address those issues if you’ve dug in heavily. I recommend analyzing GSC’s index coverage reporting today, including the powerful Excluded category. There may be some hidden treasures waiting for you. Go find them!

GG