The Internet Marketing Driver

Visualizing The SEO Engagement Trap – How To Use Behavior Flow In Google Analytics To View User Frustration [Case Study]

By Leave a Comment

When thinking about content quality, it’s incredibly important to meet or exceed user expectations based on query. If your content cannot do that, then users can end up extremely frustrated as they search Google, click through your listing in the search results, scan your page, and not find what they were looking for. And that can lead to all sorts of problems SEO-wise.

When you don’t meet or exceed user expectations, one possible result is a user jumping back to the search results to find another solution (low dwell time). That’s clearly not what you want, but beyond the jump back to the SERPs, there are also times users stay on the site and keep digging believing they can ultimately find what they were looking for. That’s a classic engagement trap where users can end up running around a site visiting page after page without finding what they were looking for.

From a broad core algorithm update standpoint, Google is not going to react well over time when sites frustrate users and cannot provide answers that meet or exceed user expectations. We’ve seen that time and time again when broad core ranking updates roll out. In addition, I’ve seen this for a very long time with major algorithm updates overall, including during medieval Panda days.

As I’ve said many times before while writing about major algorithm updates, “Hell hath no fury like a user scorned”. In other words, don’t frustrate your users or bad things can happen during major Google algorithm updates.

This sounds logical, but how you can show a site owner user frustration like I explained above? For example, real data showing that users are frustrated versus just providing your opinion. Well, I surfaced a great example recently and I’m going to share more about that below. It’s something you can do for your own site and you can accomplish this using a tool that’s freely available and right under your nose – Google Analytics.

Backing Up SEO Intuition With Data – Visualizing User Frustration
I’m helping a site now that has been negatively impacted by two broad core updates in a row. I started helping them in April after they reached out to me about being hit by the January 2020 Core Update. Then as I was auditing the site, and they were making changes, the site was hit a second time during the May 2020 Core Update.

Remember, recent changes will typically not be taken into account with broad core updates and the site was just in the beginning stages of implementing changes. You can read my latest post about the May 2020 Core Update to learn more about that point (and other important points that site owners should know about Google’s broad core updates).  

During the audit, I surfaced a number of key problems and the site owner has been moving as quickly as possible to implement those changes. One of the issues I surfaced involved user frustration and not meeting expectations based on query.

The site had a number of pages that were ranking well for queries where users had a very specific intent (and I would go as far as saying it was an extremely strong intent). Unfortunately, the pages could not help users achieve that goal. They tangentially covered the topic, but couldn’t provide exactly what users were looking for.

And beyond that, the pages contained information and links that might lead some users to believe they could still find a solution deeper on the site. This could lead to the engagement trap situation I mentioned earlier where users end up digging and digging with no possible way to find gold.

To show the impact during a broad core update, here are two pages like this on the site and their respective drops during the January 2020 core update:

As I was auditing the site, I began to experience user frustration like I mentioned earlier… I was running in circles when I put myself in the shoes of a typical user. It was extremely frustrating… and although I could see many clicks leading to these pages, GSC alone couldn’t provide the full story. So I switched to Google Analytics and a feature that I believe is completely underutilized – Behavior Flow.

Behavior flow enables you to visualize the journey users take through your site based on a dimension you select as a starting point. For example, you can view the path through your site based on traffic source, medium, campaign, landing page, and more. I bolded landing page since that’s exactly what I’m going to show you below for the page type I covered earlier.

Here is what Behavior Flow reporting looks like in GA:

A quick note about sampling:
It’s important to know that Flow Visualization reports in Google Analytics are sampled. Behavior Flow is based on 100K sessions based on the selected timeframe. To get around sampling, you can always shorten the time range to get the number of sessions under the sampling threshold. Just keep this in mind while reviewing the reporting.

When “Engagement” Could Mean Frustration: How To Use Behavior Flow To Visualize Frustrated Users
During the engagement, I ran a traffic drop report (previously called Panda report) to surface the largest drops by landing page after the January and May core updates. Again, the site had a number of key problems across various page types that were problematic. One was the page type I explained earlier, which not only couldn’t meet or exceed user expectations, but it also had the potential to lead users to more pages thinking they could find a solution.

I checked Google Analytics for landing page metrics and noticed one of the problematic urls had a 65% standard bounce rate. That means 35% of users visited another page on the site (which accounts for nearly 20K users during the timeframe). I wanted to see where those users were visiting after the landing page. In other words, were they happy with their visit or were they banging their head against their monitor while endlessly roaming around the site in search for a solution?

So I jumped to Behavior Flow in Google Analytics and set my starting point as that landing page. And once I did, I immediately noticed a problem. And that’s pretty much what I thought I would see!

For this page, 65% dropped off after visiting the landing page during the timeframe I selected (pre-core update). That’s not surprising, but the 35% remaining on the site displayed a path of frustration that was hard to ignore.

The next step for almost all of the users, which is labeled the “first interaction” after the landing page, was to pages that seemed like they could provide a solution, but really couldn’t. So, the landing page couldn’t meet user expectations and then 35% of the users landing on that page ended up visiting another page that couldn’t meet expectations either. Not good, but we’re just getting started.

And then the second interaction was eye-opening. 68% of the users that clicked to a second page went back to the original page they visited from the search results! Yep, I was beginning to see user frustration in the visuals. They visited the landing page from the SERPs, couldn’t find what they were looking for, visited another page thinking they could find a solution, but couldn’t. So they went right back to the original landing page still looking for a solution!

But the fun doesn’t stop there. The third interaction reveals users visiting even more pages from that original landing page believing they could find a solution. 20% of the users visiting the original landing page were now three interactions into their visit, with no hope of finding a solution.

But we’re not done yet. I keep using the word “trap” in this post and the fourth interaction underscores why. To quickly recap, we have users landing on a page (mostly from the SERPs), visiting another page to find a solution, then jumping back to the original landing page, and then desperately visiting more pages to find a solution.

What do you think happened next??

Yep, they went back to the original landing page again! 14% of users were now four interactions into the site (but back at the original landing page). Ugh.

And if they haven’t had enough, some users were still jumping to other pages from the original landing page to find a solution. Oh, those poor people… :)

As you can imagine, this led to a horribly frustrating experience for those users. And over time, hundreds of thousands of users were going through this process. I’m not saying this is the sole reason the site got pummeled by two core updates (remember, there are “millions of baby algorithms” at play with core updates), but it sure doesn’t help matters.

Using behavior flow in Google Analytics enabled me to visualize the frustration that users were experiencing while entering specific landing pages (and most visitors were from Google organic). And that visualization can go a long way when explaining to site owners how certain things need to change on a site. I know this example did.

I’ll end this post with some final tips and recommendations based on going through the process of visualizing user frustration using Behavior Flow in Google Analytics. Again, the great thing is you can go through this process today, and it’s free.

Using Behavior Flow To Visualize Frustration: Tips and Recommendations

  • Identify high volume pages and dig into the data. Make sure you understand what’s going on user engagement-wise. Don’t assume all visits end with happy users.
  • You can also run a traffic drop report to identify the biggest drops to landing pages across the site based on a core update, and then dig into those pages.
  • Don’t just check a metric like standard bounce rate and make decisions. Dig deeper with other user engagement metrics and with additional functionality like Behavior Flow in Google Analytics.
  • Use specific landing pages as the starting point in Behavior Flow and visualize the journey through your site. This is how you can identify engagement traps like I covered in this post.
  • If you identify engagement traps, take action before Google takes action. Always try to meet or exceed user expectations based on query. Don’t wait until you drop based on a broad core update to fix user engagement problems, content quality, and more. Identify key problems with those pages and fix them as quickly as you can.
  • Track Behavior Flow over time for the pages you identify as engagement traps. You should see improvements in the data and through the visualizations in the reporting after you fix those problems.

Summary – “Hell hath no fury like a user scorned”
This case study demonstrated how you can back up SEO intuition by using Behavior Flow in Google Analytics. By using this underutilized feature in GA, you can view the journey users take through your site. And you can use a number of dimensions as the starting point, including landing page.

Behavior Flow can absolutely help you identify problematic situations and then effectively communicate that story to site owners (backed by data). And that can help drive change. Like I said earlier, the amazing part is that it’s freely available in Google Analytics. You can literally start checking this now. So fire up Google Analytics, find those engagement traps, and fix them soon. Take action before Google takes action.

GG

The May 2020 Google Core Update – 4 Case Studies That Emphasize The Complexity Of Broad Core Algorithm Updates

By Leave a Comment

On Monday May 4, 2020 Google pushed another broad core update, which are only released three to four times per year. They are huge updates, global, and can have a significant impact for sites across categories and countries. Although the update can take about two weeks to fully roll out, you can usually begin to see impact within a few days. And we saw impact even quicker with this update. For example, we saw big swings in rankings and traffic on May 5th, just one day after the update rolled out.

In this post, I’ll first cover some key points related to core updates, since I’m hearing a lot of confusion from site owners about certain topics. These points are important for site owners to understand for several reasons and I’ll cover those below. After going through those topics, I’ll cover four case studies that provide some valuable lessons about core updates, about the need for continual monitoring and improvement, and what happens when you don’t nip SEO problems in the bud. Above all, the cases emphasize the complexity of Google’s broad core updates.

The four cases include a recovery from a previous core update, a site seeing (mostly) long-term wins during core updates, a massive drop during the May core update, and then a recovery from a non-core update during the May 2020 core update.

Clearing Up Some Confusion About Core Updates
When core updates roll out, it can be very confusing and stressful for site owners. And you can see that confusion in many of the questions that are being asked across the web, along with the questions that are sent to me privately (via email, Twitter direct messages, Facebook messages, LinkedIn, etc.)

I’ve covered core updates many times in my previous posts, so those posts always a good place to learn more about how broad core updates work. That said, I’ll organize some key points below that site owners should be aware of in order to make sound decisions about how to move forward.

1. There’s Never One Smoking Gun, There’s Typically A Battery Of Them
If you’ve read my previous posts about core updates, then you probably know this one already. With core updates, Google is taking many factors into account over an extended period of time. Based on what Google has said many times, and what I have seen in the field while helping companies that have been impacted, there’s never one or two problems that cause negative impact. Instead, there’s usually a battery of problems across the site. In addition, Google always pushes relevance adjustments, which aren’t necessarily a bad thing. More about that soon.

Regarding problems that could be causing negative impact, they span content quality, user experience, aggressive, disruptive, or deceptive advertising, technical SEO problems that cause quality problems, and more. In addition, Google is always looking to surface the most authoritative content written by experts in a niche. And that’s especially true for what Google calls “Your Money or Your Life” (YMYL) content. So on top of what I already mentioned, there could be several issues related to expertise, authoritativeness, and trust (E-A-T) which is heavily covered in Google’s quality rater guidelines and in their blog post about core updates.

Based on what I just explained, it’s not uncommon to surface many issues on a site that’s been negatively impacted, and across a number of categories. That’s why I recommend using “The Kitchen Sink” approach to remediation, which involves surfacing all potential problems on a site and fixing as many as you possibly can (all of them if that’s possible). I’ll cover this in the case studies later in this post.

2. Relevance Adjustments
Beyond the problems I listed above, there are always relevance adjustments during core updates. That’s when rankings can drop when content just isn’t as relevant anymore for a query. As an example, you can often see this happen on news publisher sites that might have an article ranking for super-competitive head terms for a given subject (like a person’s name, a company name, movie, TV show, event, etc.)

As time goes on, that article might not be as relevant anymore and rankings drop. Those types of adjustments aren’t necessarily a bad thing. It’s just important to distinguish between relevance adjustments and when you have major problems throughout a site. And you might have a combination of both (many sites often do).

For example, see the ranking changes for head terms below for a news site:

3. “Millions of baby algorithms”
Gary Illyes from Google once explained that its core ranking algorithm is made up of “millions of baby algorithms working together to output a score.” I love that quote and it makes complete sense. This is also why you shouldn’t just focus on just one or two factors. You should look at your site holistically and surface all potential problems… and then fix them all if that’s possible. That’s also why I recommend running user studies through the lens of core updates. You can learn a lot about your site when you have real people provide real feedback based on traversing your site.

4. Recovery From Core Updates
This is ultra-important to understand and it’s something Google has even confirmed in its post about core updates. If you have been negatively impacted by a core update, you (mostly) cannot see recovery from that until another core update. In addition, you will only see recovery if you significantly improve the site over the long-term. If you haven’t done enough to improve the site overall, you might have to wait several updates to see an increase as you keep improving the site. And since core updates are typically separated by 3-4 months, that means you might need to wait a while.

Here is a section from Google’s blog post about core updates:

One of the reasons it’s important to understand how this works is because some companies are implementing short-term testing of changes, and then rolling those changes back if they don’t see positive movement. That will not work, since you won’t know if those changes are helping UNTIL ANOTHER CORE UPDATE ROLLS OUT. Sure, you might see subtle changes as Google pushes many smaller updates over time, but you won’t typically see massive changes until another core update rolls out.

That means you can be spinning your wheels by testing certain things, and maybe rolling back the right changes because you don’t see short-term gains. I recommend implementing the right changes for your users and site and keeping them in place for the long-term. Significantly improve your site, your content, the user experience, and more.

5. Recent Changes Will Not Be Reflected In Core Updates
If you recently implemented a lot of changes before a core update rolled out, and either dropped or didn’t see positive impact, it’s important to understand that recent changes will often not be reflected in core updates. For example, rolling out big changes 2-3 weeks before a core update rolls out will typically not be reflected.

Google is doing a ton of evaluation over the long-term for core updates, so making a recent change before a core update rolls out is usually not the reason you saw a big drop, or even a big gain. Google’s John Mueller has explained this before and you can watch the video below to hear him explain this. Below is my tweet about this with a link to a video with Google’s John Mueller.

So again, make the right changes and keep them in place for the long-term. It’s also worth noting that Google’s Paul Haahr, a lead ranking engineer at Google, also explained to me at the Google Webmaster Conference in Mountain View that Google does a lot of evaluation in between core updates (and it can be a bottleneck that inhibits them from rolling out broad core updates more often). He also said they could potentially decouple certain algorithms from broad core updates and roll those out at a different time, but we are still looking at 3-4 updates per year in the short-term.

With those five points out of the way, let’s move on to the case studies I mentioned earlier.

May 2020 Core Update – Four Case Studies, Four Lessons
Below, I’m going to cover four cases based on sites that were impacted by Google’s May 2020 core update. Each is a unique case, with unique learnings, and my hope is site owners can learn something from each one. I can write a book about each one, but I’ll try and keep each case as concise as possible.

I’ll cover a recovery from a previous core update, a site that has (mostly) surged over time during several core updates, a major drop during the May 2020 update, and then a recovery from a non-core update during the May 2020 core update. Again, I think the cases underscore how complex core updates are, how many factors are involved, etc.

But before we jump in, there are two points I want to quickly cover. First, I wanted to thank my clients for letting me document their cases in this post. They are blinded case studies, but my clients were nice enough to let me explain more about each situation, provide data from GA and GSC, and more.

Second, I need to provide my typical disclaimer about broad core updates.
I do not work for Google. I don’t have access to its core ranking algorithm. I have been to Google Headquarters in both California and New York, and it was tempting to put a John Mueller or Martin Splitt mask on to try and speak with the search engineers, but I figured that wouldn’t end up very well. :)

Case 1 – Recovery From The March 2019 Core Update
The first case is a site that provides a unique service in the health/medical niche. The site got hammered during the March 2019 core update and dropped by 40% overnight. Like many, the site owners were extremely confused about why they dropped.

Once I started digging into the site, there were many problems I surfaced in my audit (and those problems spanned a number of topics). Remember what I said about “millions of baby algorithms” and the complexity of core updates?

Note, some of what I’ll cover below would yield an indirect effect, while other changes could yield more of a direct effect (especially when combined together). But again, the focus was to improve the site overall, and across content quality, user experience, technical SEO, and more. Google has explained many times they are looking at many factors for core updates and I think this case demonstrates that well.

User Experience
First, the site had several key UX issues, including having giant calls to action pushing down the main content throughout the site. That situation was bad on desktop, but even worse on mobile. When viewing the content on a mobile device, the entire viewport was filled with a lead capture form. I also surfaced key navigation problems on mobile which inhibited users from traversing the site via the site’s global navigation. I can’t even imagine how frustrating that could be for users.

A lead form like this one filled the viewport, above the main content:

Content Quality and Structure
Content-wise, I definitely surfaced some thin content across the site, which the site owners handled pretty quickly. This wasn’t a big problem, but was worth handling for sure. The site also combined some important content on one page that should have been on multiple pages. So instead of having specific pages covering a certain topic, they were covered on one page trying to explain it all (while visibly revealing that content via UI elements). They (mostly) fixed that and provided all of that content on their own respective pages.

Also, my client has worked hard to go back through a lot of their core content and update it, when needed (refreshing the content and enhancing it). In addition, they are now “on the pulse” of their target market and have brought in various departments that interact directly with the users of the site. That helps them build content that addresses current needs in a thorough way. Needless to say, this is extremely important when it comes to meeting or exceeding user expectations via high-quality content.

AMP, your first impression from Search
When checking GSC, you could see that many users were visiting AMP urls from the search results, so it was super-important to review those pages (usability-wise, content-wise, parity-wise, etc.) Unfortunately, the CMS was limiting with what the site could do AMP-wise and the pages were literally barebones versions of the responsive pages. The site owners did what they could do improve that experience, although it’s still not perfect. But overall, it’s a stronger user experience (and at least has stronger branding).

When filtering by AMP in Google Search Console (GSC), you can see a significant number of users from Search visiting AMP urls. For many, this was their first impression of the site:

YMYL & Authority
Next, the site is in the health/medical niche so it’s categorized as “Your money or your life” (YMYL). Any site that can “impact a person’s future happiness, health, financial stability, or safety” is held to a higher standard. Since this site publishes health and medical content, and provides a service tied to the vertical, making sure that content is written by expert-level authors was extremely important.

And beyond having expert authors produce high-quality content, making sure users (and Google) know who the authors are, and understand their experience and background, was an important change. When conducting user studies, I’ve watched, and listened to, real users comment about this. The site owners focused on making a number of key changes on this front.

From an authority standpoint, Google has explained that for assessing “authority”, the best known signal they use is PageRank (or links from across the web). So yes, the right links matter, and especially for YMYL content.

In addition, Marie Haynes asked Google’s Gary Illyes about E-A-T at Pubcon in 2018. Gary explained that it is largely about links and mentions from well-known sites. You can see Marie’s tweet below. So again, the right links matter.

Unfortunately, the site just didn’t compare well against the competition when reviewing links and mentions from well-known sites. Actually, the overall link profile was weaker in general. I brought this up several times during the audit and explained this was not an easy or quick thing to fix. Over the past year, the site has naturally gained more links, and from some important sources. This clearly could have helped the situation, especially when combined with the other changes. 

Technical SEO
There were a number of technical SEO problems as well, which spanned several categories. For example, the site’s internal linking structure was problematic, which was inhibiting key pages throughout the site from linking directly to other key pages (and that included problems with the homepage as well).

There were also canonical problems on the site, including a number of pages containing double canonical tags (each pointing to different urls). When Google sees multiple canonical tags, it can simply ignore those and select one on its own. That’s a dangerous game of “canonical roulette” to play with Google.

Last, but not least, the site owners had a full-on cage match with their robots.txt file. When starting the audit, I noticed that some important pages were being blocked from crawling. You definitely don’t want to block Google from crawling important pages, so my client removed those directives quickly. But later in the engagement, I noticed that the old robots.txt file was back! My client immediately dug in to find out what was going on, to add the new robots.txt file back, etc. Note, I’ve written several times about robots.txt issues, and this was just another case of bad things happening. Anyway, it’s just worth noting that they struggled with this problem over time. The issue has been fixed and the new file is currently running.

The Waiting Game
As additional core updates rolled out after the March 2019 update, the site would only gain or drop marginally. This wasn’t necessarily surprising since there were a lot of changes implemented from May through September and Google would need to see those changes over the long-term in order for the site to recover.

Finally, during the May 2020 core update, the site surged. It’s up 53% since the update rolled out, which is great to see. It’s not back 100% to where it was before the March 2019 update, but it’s pretty close.

Here is a snapshot of some rankings increases for the site following the May 2020 core update. There were nice gains across many different keywords:

Case 2 – From Medieval Panda to Core Updates and Beyond
The next case I’ll cover is a long-term client of mine that has done extremely well over the years after getting hit hard by medieval Panda. Since then, they have worked hard to improve the site overall, but also to continually audit the site through the lens of core updates. It’s a large-scale, complex site in the news vertical.

The site has seen nice gains during previous core updates, but has also seen some relevance adjustments during others (with some drops). The relevance adjustments make sense as older content simply isn’t as relevant for broad queries as they once were. That’s totally fine from my standpoint (and theirs) and the main focus is to maintain strong SEO hygiene over the long-term.

Again, it’s a large-scale site with many moving parts. There are hundreds of new urls published every day, the site has millions of pages indexed, and there is an international component as well.

During audits, a number of issues tend to get nipped in the bud, which helps stop those issues from becoming bigger problems. I’ll cover some things below that have been addressed as audits are conducted several times per year.

Maintaining Strong Quality Indexation
Even though the site primarily publishes high-quality content, it has always had to deal with some content that’s lower-quality or thin. So, it’s always important to surface those lower-quality urls that slip through the cracks. Remember, this is a complex site publishing many new urls per day, so it’s not simple from a quality control perspective.

Here is an example of a page with just a paragraph that didn’t cover the subject matter thoroughly. Surfacing and handling these types of pages is critically important for this site.

Technical SEO issues that can cause quality problems
Since it’s a complex site, there are times that technical SEO problems pop up, and could roll out to many pages (thousands, tens of thousands, or even more). It’s important to continually analyze the site to surface those and fix them as quickly as possible. For example, canonical problems, render problems, meta robots tag issues, robots.txt issues, performance problems, and more.

Also, with a lot of content published per day, and based on how this site works, there is always the risk of multiple urls being published containing the same, or similar, content without the correct canonical tags set up. Over time, this can yield a lot of content splitting power across multiple urls that contain the same exact content. This can dilute the power of any given piece of content, so it’s important to catch that quickly and rectify. Remember, it’s a huge site.

In addition, I surfaced a number of urls where canonical tags were pointing at urls that don’t resolve with 200s. So those canonical urls were 404s, redirects, etc. It ends up there was a coding issue where the urls in rel canonical were malformed. That’s never a good thing, especially on a large-scale site with millions of pages indexed.

Also, there were tens of thousands of urls listed as “Discovered, not indexed”, which as John Mueller has explained, can signal quality problems or even crawl budget problems. Analyzing those urls, findings patterns, and addressing those issues was important.

Google’s John Mueller about “Discovered, not indexed”:

As you can see, there are many things being surfaced during audits that are conducted throughout the year (even beyond what I’ve covered here today). For this site, maintaining strong SEO hygiene has helped it increase in visibility over time, while seeing surges during several core updates (including the May 2020 core update).

Authority – The importance of A in E-A-T
Beyond content and technical SEO, the A in E-A-T is very important (and I covered that earlier in the post). This site is definitely seen an authoritative source of news. Their content is often referenced from some of the most powerful sites on the web. I’m sure this is helping the site with core updates, especially for news coverage.

Based on the May update, the site is up 35% accounting for an additional 40-60K sessions per day from Google organic. This is after surging during the September 2019 update.

Case 3 – The Ghost of Fred And A Reminder To Continually Analyze
Next up is a site that has continually battled major algorithm updates over time, has recovered from several, including a massive surge during Fred, but just got hit hard during the May 2020 update. It’s in a tough niche, since it contains the same, or very similar, content to many other sites. I helped the site in the past and have been in close contact with the site owners since then. The site hasn’t been audited in quite some time and they quickly reached out to me after seeing a significant drop in rankings and traffic based on the May core update.

Tough Niche and The Need To Clearly Differentiate Your Site
Google’s John Mueller has covered the topic of having the same, or similar, content as many other sites several times over the years. He has explained that you need to differentiate your site as much as possible from the rest. If not, it’s very hard for Google to determine which site should rank well over the others.

Here is my tweet about the subject with a link to a video of John Mueller explaining more about the situation:

For this site, it’s a case where it has not been audited on a regular basis and several problems have crept in. And over time, a number of important issues have riddled the site. And the result during the May core update was disastrous. Google organic traffic is down 40% since the update rolled out.

Aggressive and Disruptive Advertising
This is a problem that the site has struggled with over the years and it’s pretty aggressive again.  The aggressive and disruptive advertising situation inhibits the user experience across both desktop and mobile. Ads are pushing the main content down the page, sometimes double ads are delivered in certain sections of the page, some are auto-playing video, and more. I’ve written many times about the danger of aggressive monetization tactics, and even Google has explained this in their blog post about core updates.

Quality Indexation
From a quality indexation standpoint, there is a lot of thin content across the site based on the niche. For example, thousands of pages that contain very little content, or even no content on some pages. In addition, there are also thousands of pages that shouldn’t be indexed at all that are currently being indexed. They aren’t the canonical versions of the pages, but are still being indexed. Those should definitely be nuked.

Here are some examples of thin content from the site. There are thousands of pages like this across the site:

Separate Mobile URLs – Lack Of Parity
In addition, there are separate mobile urls and those pages do not match the desktop pages across several key elements including links, navigation, and some content. As you can guess, the site has not been moved to mobile-first indexing yet and most of their traffic is via mobile.

So, the site has an aggressive ad situation mixed with mobile problems on a site that contains the same, or very similar, content as many other sites on the web. Then add a thin content problem and that makes the situation even worse. It’s not a great recipe.

The site owners are already working on changes, which is great. The tough part is they will have to wait for another core update to see recovery (or possibly multiple core updates).

Case 4 – Recovery From A Non-Core Update During The May 2020 Core Update
The last case I’ll cover is the most interesting one in my opinion. The site got smoked by an unconfirmed update on April 2, 2019, and when I say “smoked”, I mean it was hit really, really hard… Check the trending below from GSC and GA:

It’s an affiliate site that got blindsided and dropped by a whopping 44% overnight. The site owners reached out to me soon after, but I couldn’t start helping them until the summer of 2019. That April update wasn’t a big one overall (there weren’t many sites impacted across the web), but there were some I found (with very similar extreme drops). Also, some of those were affiliate sites as well. Just an interesting side note.

First, the site now produces high-quality, and helpful, content about the products they are covering. So, if you checked much of the site’s current content, you would definitely say it’s high-quality content. But the devil is in the details and why it’s always important to heavily, and thoroughly, analyze a site. That’s when you can start to surface problems that might not be apparent at first glance.

Aggressive Affiliate Setup
First, it was hard to ignore the aggressive affiliate situation on the site. The pages had large and disruptive affiliate modules weaved throughout each page. And some were located at the top pushing the main content down the page (especially on mobile).

I asked my client more about the affiliate situation, and some interesting bits of information came from that conversation. For example, it ends up they implemented a much more aggressive affiliate setup in January, just a few months before the April drop. Therefore, the site made a clear move from an acceptable user experience to an aggressive and disruptive one (in my opinion).

Lower-Quality/Thin Content
Next, I did surface pages that were thin or lower-quality. This wasn’t a huge problem for the site, but it was definitely worth addressing. As Google has explained before, every page indexed is taken into account when evaluating quality.

For example, there were some articles that were only a paragraph or two in length that didn’t thoroughly cover the subject. Also, Google’s John Mueller has explained several times that affiliate sites are fine from Google’s standpoint SEO-wise, but affiliate sites with low-quality or thin content are NOT fine (and that’s usually the problem when they experience drops in traffic).

Here is a thin article I surfaced during the audit:

Cutting (Redirect) Ties and Killing Mutual Links
I can’t explain too much on this front, but there is another site they own that covers a different vertical. There were thousands of links from that sister site to this one and many were followed. There were also redirects set up from a number of older urls on that site to the one I’m covering here. Also, both sites were hit on 4/2/19. Yep, it was an interesting case for this company.  

So, the site owners finally made the decision to remove all ties from that second site to this one. All links and redirects were removed during the engagement. Hard to say how much this impacted the situation, but this site is surging during the May core update, and the other site has increased as well. It’s just an interesting side note.

Example of links from the sister site to this one. All were nuked, including all redirects from the sister sister site to urls on this site:

Nuked Comments
Since user comments are seen as part of the overall page content, they can drag quality down if they aren’t heavily moderated and high-quality. That’s why it’s important to know how Google is handling your comments from a rendering and indexing standpoint. Based on the current state of commenting on the site, in addition to technical problems with the comments, the site owners decided to nuke all comments from all pages.

E-A-T
The site has naturally built some very strong links over time, which again, can help greatly with building authority. Over the past year, they have continued to build links and mentions from well-known sites. Like the other examples above, I’m sure this has helped their situation.

Also, the site has strong writers that cover the niche, so making sure users, and Google, understood who was writing the content was important. Several changes were implemented on that front.  

And beyond what I mentioned already, I was surfacing a number of technical SEO problems along the way. The site owners were fixing those quickly as I surfaced them.

I can’t cover everything that was surfaced and fixed, but I think you get the picture. There were a number of important problems that were riddling the site. One of my emails to the site owners summed up the situation nicely (written during the engagement). I’ve included that paragraph below. It’s important to understand that there’s never one smoking gun. Instead, there’s typically a battery of them. This was a great example of that:

“It could very well be multiple issues working together that could be causing problems. E.g. aggressive affiliate marketing + thin content + technical SEO issues causing quality issues + everything else I’ve sent along. And… then what I haven’t sent yet. I’m not done analyzing the site.”

So the site owners moved to fix as much as they could over time. They definitely implemented many of the changes I recommended during the engagement. Since the site had been impacted by a non-core update on April 2, 2019, we were eager to see how it was impacted during future core updates. Then the May core update rolled out and the site started to surge. It was great to see.

The site is up 36% since the update started rolling out and their rankings have shot up for many keywords across categories. And that includes gaining a number of featured snippets (which can happen when you are rewarded during a core update). They aren’t back to where they were before April of 2019, but they will take a 36% increase for sure. :)

I have provided GSC and GA data below, in addition to a snapshot of rankings that increased after the update:

Core Updates: Some Final Tips and Recommendations

  • “Millions of baby algorithms” – Remember, Google’s Gary Illyes explained that its core ranking algorithm is made up of “millions of baby algorithms” working together to output a score. Don’t try and look for that one thing that’s causing a problem. Surface all potential problems and fix them all, if possible. Use the “kitchen sink” approach to remediation that I’ve covered many times.
  • Objectively analyze your site. And if you can, run a user study through the lens of core updates. I wrote a case study about that last year that explained how it works, what you can learn, etc. And don’t just have a few people close to you go through the site (like your spouse, kids, or coworkers). Have objective third-parties review the site and give real feedback. That’s how you can improve.
  • Understand the key points about core updates that I covered earlier regarding recovery, recent changes not impacting core updates, Google looking to see long-term, significant improvement, and more. This is definitely not a “band-aid” situation. You need to dig deep and tackle big issues on the site from content to UX to advertising to technical SEO, and more.  
  • Google’s Documentation – Read Google’s blog post about core updates and then read Google’s quality rater guidelines (QRG) several times. The blog post contains a lot of good information about core updates directly from Google, while the QRG contains 168 pages of SEO gold. The QRG doesn’t map one to one with Google’s algorithms, but it sure gives you some amazing pieces of information about what Google deems low versus high-quality.
  • Don’t roll back changes too fast. I explained earlier that you can only recover from a core update during a subsequent core update (as confirmed by Google and based on what I’ve seen many times while helping companies). Implement the right changes for your users and site, and keep them in place over the long-term. If not, you might roll back the right changes after not seeing short-term gains (and miss the long-term gains you could have seen during future core updates).

Summary – Core updates are complex.
I hope it was helpful to read through the four case studies I covered in this post. I think they demonstrate the various ways in which a core update can impact a site, the types of problems that can potentially cause negative impact, and what sites can do to recover.

And above all, the cases demonstrate how complex Google’s core updates are. So, if you have been negatively impacted by the May 2020 core update, I would begin to objectively analyze the site soon. And then work to significantly improve the site over the long-term. That’s exactly what Google wants to see. Good luck.

GG

How To Remove An Image From Google Search Using The Outdated Content Tool (When The Image Was Published On Another Site)

By Leave a Comment

How to remove images from Google.

There are times that pesky image search results stick around on Google even when those images have been removed from the web (or at least you think they have been removed from the web). I’ve received many calls over the years about this very situation. And many times, the images that people want to remove are on third-party sites outside of that person’s control. For example, profile photos from Twitter, Facebook, images from articles that have been removed, etc.

Since the sites aren’t under your direct control, your options are limited. For example, for times you can’t directly access the third-party site and you can’t access the Google Search Console (GSC) property for the site to use the Removals tool. From an image search standpoint, it can unfortunately lag behind web search for updating and/or removing files. And that can result in images hanging around the search results when they really should be removed, which can be extremely frustrating.

So what can you do??

Well, as long as the image has been removed from the third-party page that contained it, or if the page has been entirely removed from that site, Google provides functionality for quickly removing that image from the search results. On that note, I find there’s a lot of confusion about the various tools Google provides for removals, and how to use them. I’ll explain more about that soon.

For this post, I’m going to show you how to remove an image from Google Images, when that image resides on a third-party site (out of your control). I’ll walk you through using the Outdated Content Tool to target those images and remove them from the image results.

The Removals Tools in GSC Versus The Outdated Content Tool
First, I know many people aren’t familiar with Google’s Outdated Content Tool, which has been around for a long time. It’s a publicly available tool that you can use to request removal of webpages or images (as long as that content has actually been removed). I have used this tool many times over the years when helping clients. And it’s important to understand that it’s different from the Removals tool in GSC.

It’s also important to note that the content you are attempting to remove from the search results must have been removed already from the third-party site for this tool to work. You cannot simply log into the tool and start nuking content from another site when that content still exists.

In the tool, you can request new removals (e.g. webpages or images) and you will also see previous requests and the status of those requests.

Again, this is different from the Removals Tool in GSC, where you can temporarily hide content from the search results from sites you control for six months (while you actually remove the content on your site). And if the content is removed behind the scenes, it will remain out of Google’s index, and out of its search results long-term (beyond the six-month temporary removal).

Adding More Confusion To The Removal Tools Situation
The new Removals Tool in GSC will also show you “outdated content removal requests” (which were submitted via the public Outdated Content Tool I mentioned earlier! So as a site owner, you can see any content that someone has attempted to remove on your site using the public tool.

Note, I would check this report soon… since you might be surprised what you find there. For example, I’ve seen people try to remove important files or directories from sites they don’t control using the tool. Scary, right? Of course, since the content was still present on the site, Google declined the removal request and nothing happened. But it’s still important to know what’s going on there.

For example, here is a screenshot showing three submitted requests flagging my client’s content as “explicit”. All three were denied by Google, but scary to think there was an attempt like that:

How To Remove An Image From Google Images Using The Outdated Content Tool
Now that I’ve covered the various removal tools from Google, I’m going to walk you through removing an image from Google Image Search. If you have an actual image you want removed, then now is a great time to use that image while following along.

Google provides a help document listing the necessary steps and I’ll walk you through those steps below (along with any tips I can provide along the way):

1. Go to Google Images and search for the image you want to remove.

2. From the search results, right-click on that image and select “Copy Link Address”. Don’t click the image… just right-click directly in the search results when you find the image.

3. I recommend creating a text file that contains the urls you want to remove. There may be more than one url when it’s all said and done and it’s best to store them in a file for future reference. Paste the url you just copied from the image search results into that text document.

4. There are times one image ends up being multiple images. That’s because some sites keep images in multiple sizes or at multiple urls. So, you want to make sure you copy all of those image urls for removal. If not, as one image is removed from the search results, another might take its place.

You can find other urls for the same image by right-clicking on the image in the search results and selecting “Search Google for Image”. If there are other sizes, click the links to view the image results for the other sizes. Then right-click and copy the urls like we did earlier (directly from the search results).

5. Go to the public Outdated Content Tool and paste the urls one by one into the tool and click “Request Removal”. You will need to enter each url separately. Note, I’m not actually submitting the form to remove the image from Google. It’s just an example for the tutorial. :)

Depending on the type of removal, you will see a different message. For example, if it’s just the image you are removing, and the page has been removed on the third-party site, then you will see a message explaining that the image is “no longer live on the website”. If it is gone, you can continue by clicking “Request removal”.

If you are removing an image that has been removed from a page, but the page still remains, you will see a different message. If the image has indeed been updated or removed from that page, then you should click “Yes” in the form and continue. And in case you’re wondering, the webpage isn’t going to be taken down… it’s just the image that will be purged. That’s what you want.

Congratulations, you have successfully requested removal of an image from Google Images! Well, sort of… Now you need to let Google do its thing and remove the images from Search. More about that next. You will see the status of “Pending” until the images are removed.

How Long Does It Take Before The Image Is Actually Removed? It’s Pretty Quick
When you request removal of an image, Google can work really fast. I’ve seen images removed within a day or so. You can always check the Outdated Content Tool to see the status of the removals. And you can obviously check Google Images to see if the images are indeed gone from the image search results. When the images are removed, you will see the status change from “Pending” to “Removed”.

And you will also see the type of removal requested. The status could be “Outdated page removal”, which is when the entire page has been removed, or “Changed content”, when the just an image has been removed from a page that still exists.

It’s important to note that when you are removing content from a page that still exists, Google will first purge the snippet and the cache for that page. Then the cache and snippet will be refreshed the next time the page is crawled. If you want, you can run a site query on the page and you will see the listing without a snippet and cache link.

For example, enter site:{enter page url here} in Google. You should see the snippet has been removed and the cache link is gone until Google recrawls the page.

Summary – Removing Images From Google That Are Caught In The Matrix
Having an image linger around Google Images when it has really been removed can be very frustrating. But as you can see, there are ways to remove those images from the search results (from third-party sites) by using the tools Google has provided. And once you submit your removals, it only takes a day or so for the images to be removed from the results. But remember, make sure the images have really been removed from the web. If not, the request will not be successful. 

GG

Chrome’s Timeline For Blocking Mixed Content – Examples of Blocking, How SEOs Can Test Pages, and How To Prepare For Mixed Images in Chrome 81

By Leave a Comment

In October of 2019, Google announced a plan to gradually block mixed content over time. Mixed content is a common issue across sites where secure pages attempt to load insecure content. For example, loading an image file over http from an https page. There are security risks associated with that, so Chrome wants to eliminate that risk in the future (by blocking mixed content from loading).

Since it’s a common issue, Chrome announced a staged plan in October so site owners would have plenty of time to research and fix mixed content issues on their sites. Even though the blog post was published in October of 2019, I’m still seeing a lot of confusion about the topic when working with clients. In addition, I’m finding many companies don’t even know that Chrome has begun blocking some types of insecure content already. 

In this post, I’ll first cover the timeline that Google mapped out for Chrome’s handling of mixed content. Then I’ll cover how the problem could break pages visually, I’ll cover the rendering impact, I’ll provide examples of what blocking looks like, and then I’ll end with some final tips.

Chrome’s Mixed Content Timeline:
In Google’s blog post about blocking mixed content over time, they provided a staged rollout. The plan was to gradually block mixed content versus simply flipping a switch and having pages break.

  • The first change was in Chrome 79, where Chrome started providing a setting where users could unblock mixed content on specific sites. The setting applies to mixed scripts, iframes, and other types of content that Chrome blocks by default.
  • In Chrome 80, which is the current version of Chrome as I write this article, Chrome will attempt to auto-upgrade audio and video resources to https (if possible). If the resources cannot be auto-upgraded, then Chrome will block the resources from loading. This is completed automatically by Chrome for sites trying to load audio or video resources over http from an https page.
  • In addition, Chrome 80 will allow mixed images to load, but that will cause a “Not secure” label to be displayed in the browser. So, loading images over http will still allow the images to appear on the page (as of now), but the url will have a “Not secure” label in the browser window.
  • Then in Chrome 81, which was supposed to be released in February 2020 but hasn’t rolled out, Chrome will attempt to auto-upgrade mixed images to https. If they fail to load over https, then Chrome will block those images completely (they won’t load on the page).
  • Regarding the rollout of Chrome 81, the Chrome team announced that based on the global pandemic, Chrome 81 will not be released yet. Therefore, the final stage in the gradual rollout of blocking mixed content is not live yet. But, you can test it out via the Canary build of Chrome (which I’ll cover soon). As of today (Chrome 80), the stable version of Chrome is not auto-upgrading or blocking mixed images yet. It’s just showing a “Not secure” label for mixed images. But Chrome 80 is auto-upgrading audio and video resources to https and blocking them if they won’t load over https.

Insecure Images, A Common Mixed Content Issue
I’m sure some site owners and SEOs reading this post are confused about what mixed content is and how it looks. First, if you are running your pages over https, then all resources should be loaded over https. If you try to load resources over http, then that’s where mixed content warnings arise. Again, it’s a common issue that’s impacting many sites across the web.

As a simple example, here is a secure page (https) that is loading three images over http (insecure) in the stable version of Chrome (v80). Notice the “Not secure” label in Chrome versus the secure padlock.

When using Chrome Dev Tools and checking the security tab, you can view the resources that were loaded over http. You typically need to refresh the page to see those resources.

Since we are in stage three of the rollout (Chrome 80), the images will be allowed to load, but Chrome will display a “Not-secure” label in the browser window. To clarify, the images will still show up on the page and you will not run into any rendering issues (yet).  

It’s worth noting that audio and video will be auto-upgraded to https in Chrome 80. If they can’t be loaded over https, then the content will be blocked entirely. I’m focusing on images in this post, but that’s important to know if you are using audio and video on your pages.

Chrome Canary – The Ability To See Into Chrome’s Future
The Chrome Canary build is a version of Chrome that’s updated nightly with the latest functionality. It’s a great way to test your site and functionality for future versions of Chrome. There are many times I’m activating functionality in Canary that’s not yet in the stable version of Chrome to see how that will work when it finally does roll out. 

Since I know that Chrome 81 is nearing release, I was able to test how Chrome would handle mixed images. Remember, the stable version of Chrome simply adds a “Not secure” label in the browser window, but Chrome 81 will auto-upgrade those mixed images if possible. And if the images cannot load over https, then they will be blocked entirely. That means they won’t load on the page. And that also means it can impact what Google sees when it renders your pages.

As a quick example, I set up a page that loads an image via http and that image cannot be auto-upgraded to https. As you can see below, the image is blocked entirely from Chrome (and doesn’t show up on the page). The page actually shows up as secure in Canary (the “Not secure” label has been removed), but the page simply displays a missing image icon. That’s not good if it’s a critical piece of content for users.

Image blocked from loading:

The page is now secure, but the image is blocked:

The resource failed to load since it can’t be auto-upgraded:

Evergreen Googlebot, Broken Renders, and Ranking Impact
OK, so now we know that mixed images will not load in Chrome 81 if those images cannot be auto-upgraded to https. And we saw how that looks in the browser with broken image icons showing up on the page. That’s not good for users at all, but what about Google?

Google’s John Mueller has explained that since Google uses an evergreen Googlebot now for rendering, it will be using the latest version of Chrome. That means it will be using Chrome 81 once that rolls out. Again, that hasn’t rolled out yet due to the global pandemic, but it will at some point soon.

So if Googlebot will be using Chrome 81 to render your content, and we already know that Chrome 81 will block mixed images, then that’s exactly how Google will see it. So key images will be missing from your renders if those images cannot be loaded over https.

In addition, from a ranking perspective, John explained that the blocking of images shouldn’t impact rankings based on Google’s https ranking signal. That’s good to know, but he also explained that if the pages cannot render completely in a browser, then Google probably won’t be able to render those parts either. And he clearly recommended fixing that issue.

Here is a link and screenshot to a reddit thread where John explained this:

Wrapping Up and Final Tips:
Before I end this post, I wanted to provide a number of final tips and suggestions based on researching this topic. Remember, site owners still have time before Chrome 81 rolls out to address mixed content issues. Based on what I provided in this post, it would be smart to rectify those issues now before Chrome 81 rolls out. Here are some final tips:

  • Crawl your site and surface all mixed content issues. My three favorite crawling tools are DeepCrawl, Screaming Frog, and Sitebulb. All three provide specific reporting for mixed content. Note, I’m on the customer advisory board for DeepCrawl. It has been one of my favorite enterprise crawling tools since 2013.
  • Understand that Chrome 81 has NOT rolled out yet and Google has paused updates due to the global pandemic. Chrome 80 is the stable version of Chrome now and it does display a “Not secure” label for pages with mixed content problems. Also, Chrome 81 will roll out relatively soon and will completely block mixed images that don’t load over https (when auto-upgraded).
  • Chrome 80 currently attempts to auto-upgrade audio and video to https. And it those resources fail to load over https, Chrome will block them from loading.
  • Mixed content problems can impact your pages visibly for users, and for Google rendering your pages. Remember, Google now uses an Evergreen Googlebot, which utilizes the latest version of Chrome to render your pages. That means Googlebot will be using Chrome 81 to render pages once it rolls out. As Google’s John Mueller explained, you definitely want to fix mixed content issues that impact rendering.
  • John Mueller also explained that this shouldn’t impact rankings based on Google’s https ranking signal. That’s good news, but again, it can impact what users see on your pages and what Google is able to render.
  • You can use Chrome Canary to test your pages now to see how mixed content will impact your site once Chrome 81 rolls out. It’s a great way to test Chrome’s latest and greatest functionality that hasn’t rolled out to the stable version of Chrome yet.

Address Mixed Content Issues and Avoid Broken Pages and Partial Renders
In this post, I covered Chrome’s plan to gradually block mixed content. The final stage of the rollout will be completed when Chrome 81 is released as the stable version of Chrome. But as I covered in my post, the release of Chrome 81 has been delayed due to the global pandemic. Therefore, you still have time to fix mixed image problems before Chrome blocks them from loading (if they can’t be auto-upgraded to https). And remember, audio and video resources are already being blocked if they can’t be auto-upgraded.

I recommend thoroughly testing your site to ensure you surface, and then fix, mixed content problems. Then you can rest assured that both users and Googlebot can load and render all of your resources, including images, video, and audio.

GG

Revisiting The SEO Importance of Quality Indexation For Large-Scale Sites: Hunting High Indexation and Low Search Value By Directory [Case Study]

By Leave a Comment

Quality Indexation for SEO

A Quick Reintroduction To Quality Indexation:
I’ve covered the importance of what I call “quality indexation” many times in my posts about major algorithm updates. It’s an extremely important topic, especially for larger-scale sites. When I refer to quality indexation, I’m referring to the importance of making sure your highest quality content gets indexed, while ensuring your low-quality or thin content remains out of the index. With Google confirming that it takes all pages indexed (10:06 in the video) into account when evaluating quality (25:20 in the video) for a site, it’s important to make sure your ratio is strong.

I see weak quality indexation rear its ugly head often while helping companies that have been negatively impacted by Google’s core algorithm updates. For example, there may be sections of a site containing massive amounts of indexed urls that are thin or low-quality. So the ratio of high quality content to low-quality content is not strong.

And when checking traffic to those areas from organic search, you don’t see much. So, you have massive indexation with low traffic levels from Google organic. That’s not a great combination for site owners and SEOs. I’ll provide more information about this soon.

In addition, this situation can cause crawl budget issues. Note, most sites (like 99.9%) don’t need to worry about crawl budget. But larger, more complex sites often do need to worry about it. For example, a large-scale site with millions of urls, and a complex setup, should definitely make sure crawl budget isn’t an issue. So, we have quality and crawl budget concerns when quality indexation is weak.

For example, here’s the coverage reporting for a large-scale site with millions of pages indexed and tens of millions of pages excluded. It’s extremely important for this site to focus on strong quality indexation:

Crawl budget can be an issue for large-scale websites.

In worst-case scenarios, there may be millions, or tens of millions, of low-quality urls indexed that can end up dampening the quality of the site overall in Google’s eyes. I am working with a client now where this is the case. The good news is that we are actively working on fixing the situation once I uncovered the problem. The bad news is that this has been in place for a long time and Google will need to figure the changes out over time across tens of millions of urls. That said, the site owner wants to get the site in order for long-term success and understands this is the right way to proceed.

In this post, I’ll explain a process for identifying weak quality indexation across a large-scale site by directory. It’s a strong approach, especially when you’re dealing with a large and complex site.

Pre-Audit Analysis In GA and GSC:
Before fully digging into an audit, it’s always a good idea to spot-check certain reports in GSC, review the search history of the site, and more. You can sometimes find glaring issues that give you a hint about what’s going on with a site and why it has lost rankings.

For large-scale sites, GSC’s coverage report is gold. It reports on errors, pages indexed, and then contains an entire section for urls excluded from indexing. I’ve mentioned the coverage report many times in my posts about SEO and I can’t emphasize enough how powerful the reporting can be.

And when you combine the coverage report with data from the performance report, you can often identify problematic areas of the site from a quality indexation standpoint. For example, are there areas of a site with millions of pages indexed, yet those pages receive very little traffic (or no traffic)? If so, why?

The Performance and Coverage reports in Google Search Console (GSC)

And when checking the performance report, you can cross-reference Google Analytics data to see overall traffic to a page (or section) and isolate Google organic traffic. That can also help you figure out the importance and value of certain page types.

And if it’s a large-scale site with millions of urls, are xml sitemaps in order? How does indexation look when reviewing the coverage reporting for those sitemaps specifically (which you can do in GSC).

Sitemap error in Google Search Console (GSC)

Google has explained that sitemaps are extremely important for larger-scale sites. For example, Google’s Gary Illyes said sitemaps are the second most-important source of URL discovery. In addition, John Mueller has explained that sitemaps can help Google understand canonical urls, when those urls change (by providing an accurate lastmod date per url), and more.

Google's Gary Illyes explaining that xml sitemaps are the second most important source of URL discovery.

And all of what I mentioned above can help you answer a number of important questions. For example, should the pages be indexed or are they bogging down quality for the site? Are people even searching for topics that those pages could address? And can those pages meet or exceed expectations?

This process can often yield many pages that can be improved, or nuked from the site by either noindexing or 404ing the urls in question (if they simply shouldn’t be indexed). You can also find urls that should be canonicalized to other urls if they contain duplicate content. All of this is important to know so you can take action.

The Importance And Power Of Adding GSC Properties *By Directory*:
OK, so you now you’re excited to jump into GSC’s coverage report to begin some detective work. Well, I have some bad news for you. First, the coverage report will cover the entire site, so you can’t slice and dice your data by directory. Second, the reporting is limited to just one thousand urls per report. That’s clearly not sufficient when dealing with a site that contains millions of urls.

Note, I’ve repeatedly asked Googlers for an index coverage API so you can export all urls by report. I even did this in person at Google’s webmaster conference where I spoke with the GSC product team directly! I’m sad to say that I don’t believe it’s coming any time soon, so let’s move on.

So what’s a site owner to do when there are limitations like that?

Well, there’s a hack that works pretty darn well. And it can help you isolate problems per directory and gain a better understanding for quality indexation levels across each directory. The solution is adding properties to GSC for each directory. I’ve written a few blog posts about this topic and I always tell clients with larger-scale sites to set this up ASAP.

After adding each important directory to GSC, you can view the coverage and performance reporting JUST FOR THAT DIRECTORY. You can get a better feel for that directory by drilling into the various reporting without having to worry about the other directories muddying up the data.

Google's documentation for adding directories as properties in Search Console (GSC)
Adding a directory to Google Search Console (GSC) as a property

Case Study: Drill Into Directories And Let The Data Guide You
Back to the case study. Earlier I explained that I’m helping a client with a pretty serious quality indexation problem and it’s a great example of what I’ve explained so far in this post. By using the methods I covered, it was easy to identify problematic areas of the site where many pages were indexed, yet those areas received almost no organic search traffic.

And I’m not talking about a few thousand pages indexed here or there. I’m referring to millions of pages indexed with almost no organic search value at all. That helped us identify clear candidates for deindexing (either via noindexing, 404ing, or by canonicalizing urls that were duplicates).

Below, I’ll cover some quick examples of high indexation and low search value. I think you’ll get the picture pretty quickly. Let’s begin.

Directory One – Weak Quality Indexation
The directory contains pages targeting broad queries that this site will never rank for. It’s not even the main focus of the site. There are 25.3K pages indexed according to GSC, yet only 28 clicks. Yep, just 28. GA shows very low levels of traffic from Google organic as well.

Quality indexation problem example one.
Quality indexation problem and low traffic from Google organic.

Directory Two – Weaker Quality Indexation
The next directory that raised red flags yielded even weaker quality indexation levels. It also covered an area of content that was fine to have for users once they are on the site (it was tangentially relevant to the main topic for the site), but the content wasn’t great and many people wouldn’t even be searching for this type of information. There are 125K pages indexed in the directory, yet only 39 clicks in GSC.

Quality indexation problem example two.
Low traffic levels from problematic areas of a site.

Directory  Three – Duplicate content not handled properly.
The third directory I checked contained mass-duplicate content that was not being handled properly. For example, the urls should be canonicalized to the true canonical urls. There were 491K pages indexed in this directory, yet there were only 224 clicks during the time period.

Quality indexation problem example three.
Low number of clicks from Google organic to problematic areas of a website.

Directory Four – This is getting scary.
The next directory that I had concerns with had a whopping 3.04M pages indexed. It yielded more traffic than the previous three I checked, but still only 5.79K clicks. That’s not very much based on total indexation for the directory. Not good.

Quality indexation problem example four.
A low ratio of traffic to indexed pages for a problematic area of a website.

And with each additional directory I checked, I saw more of the same. Many pages indexed that weren’t yielding any (or much) organic search traffic. This was a great example of how weak quality indexation could bog down a site.

Moving Forward: Taking Action To Improve Quality Indexation
I’m working with the site owner to fix this situation as quickly as we can. Since it’s a huge site, this will take time to fix and then time for Google to reprocess the changes. Our goal is to make sure only the highest-quality pages are indexed while ensuring low-quality, thin, or duplicate content is handled properly from an SEO standpoint.

Below, I’ll end this post with some final tips and recommendations for enhancing quality indexation. Again, it’s an incredibly important topic for SEO.

Some tips and recommendations when investigating quality indexation problems:

  • First, if you manage a large-scale site, make sure you focus on thoroughly understanding quality indexation levels across the site (by directory if you can). Don’t end up with many low-quality or thin pages indexed.
  • As I covered earlier, definitely add GSC properties by directory if you can. That will enable you to isolate those directories and view all GSC reporting by that subfolder.
  • Understand the organic search value for each directory. You can leverage both GSC and GA data to verify traffic per site section and compare that to indexation per directory. And you should crawl those areas to fully understand what’s going on there from a content standpoint, technical SEO standpoint, etc.
  • Objectively analyze pages from each directory to understand the true value for the user. Ask yourself what would users search for to find that content? And if they find the content via Search, would it meet or exceed user expectations? If you’re having a hard time answering those questions, the pages could be good candidates for deindexing.
  • Form a plan of attack for improving quality indexation across the site. And understand volatility will probably come along for the ride if you are implementing significant changes across a large-scale site. That said, do what’s best for the long-term health of the site.
  • Track changes the best you can by documenting your coverage reporting and search analytics reporting over time. Coverage reporting is just for the past 90 days while search analytics data is for the past 16 months.
  • Don’t forget to set up accurate and thorough xml sitemaps (especially for larger-scale sites). They can help on multiple levels as I mentioned earlier. And if you are running a large-scale site, use sitemap index files to make sure you have full coverage of your site’s canonical urls. Make sure lastmod dates are accurate and are updated whenever content significantly changes across pages.

Summary – Don’t let weak quality indexation drag down your site.
In this post, I documented a good case of a site dealing with quality indexation problems. And with Google confirming that every page indexed is taken into account when evaluating quality, having millions of additional low-quality or thin pages indexed is not a good thing.

If you manage a large-scale site, this can become especially dangerous SEO-wise. I recommend going through the process I documented above and surfacing quality indexation problems. Then you should address those problems efficiently to enhance the long-term health of the site. You might experience some volatility in the short-term, but your goal should be long-term SEO success. I would focus on that. Good luck.

GG