The Internet Marketing Driver

Smart Delta Reports – How To Automate Exporting, Filtering, and Comparing Google Search Data Across Timeframes Via The Search Console API and Analytics Edge

By Leave a Comment

How to automate a delta report via Analytics Edge in Excel.

In 2013, I wrote a post explaining how to create what I called a Panda Report, which enabled you to identify landing pages seeing the biggest drop during a major algorithm update. The post explained how to do this based on Google Analytics data, but you can definitely do the same thing with GSC data (and with queries in addition to landing pages).

Well, it’s 2021, the process I use has been enhanced, and I wanted to publish a new post explaining how to automate the process using Analytics Edge. First, since medieval Panda is now missing in the SEO trees, the report needed a new name. For the past several years, I’ve simply called it a Delta Report. That fits much better since you can use this approach to identify the change in impressions, clicks, or rankings based on any event impacting Search (like a broad core update, a site migration, website redesign, or any other situation causing volatility). By identifying the landing pages and/or queries seeing the biggest drop, you can often find glaring issues. It’s a great way to start digging into the data after a big drop or surge in rankings and traffic.

And beyond the name change, there are some great ways to bulk export data via the Search Console API now. Since the GSC UI limits exports to just one thousand rows of data per report, using the Search Console API is critically important for exporting all of your data. For example, I often use the API to mass-export landing pages and queries from GSC (going well beyond the one thousand row limit).

A Delta Report is a Panda Report on Steroids – The Power of Analytics Edge and APIs
I have covered Analytics Edge a number of times before in blog posts about exporting data. It’s an add-on for Microsoft Excel that quickly and efficiently enables you to bulk export data via the Search Console API, Google Analytics API, and more. But that’s not all you can do with Analytics Edge. Beyond just the export, you can create macros that filter and organize the data to create advanced reports (so you can export, compare, filter, etc. and all in one shot).

For our purposes, that means exporting all queries or landing pages, comparing the data based on timeframe, filtering based on your site structure, important query types, etc., and then writing those exports to their own worksheets for further analysis. And this is all done in one shot based on the macro you create (build once, use often).

Excited? Below, I’ll walk you through the process of creating the macro via Analytics Edge. And while you go through this tutorial, you’ll probably think of 100 other things you can use Analytics Edge for while working with data. It’s basically a Swiss Army Knife for working with APIs.

How to export GSC data via the Search Console API, compare data across timeframes, filter the data by page or query type, and create separate worksheets, all in one shot:
First, there are some requirements. Although Analytics Edge isn’t free, it’s extremely cost-effective. The core add-in costs $99 per year and the GSC connector costs $50 per year. Also, the good news is that there is a 30-day free trial so you can walk through this tutorial and use the process for 30 days to see how it works for you.

So, for $150 per year, you can use Analytics Edge to your heart’s delight. If you are helping larger sites where the API is necessary to export all of your data, then Analytics Edge is a great way to go, and it definitely won’t break the bank.

How To Create A Delta Report: Step-by-step instructions

1. Download and Install Analytics Edge (Core Add-in):
You can download and run the Analytics Edge installer to quickly install the add-in. After installing the add-in, click the license button and accept the Terms of Use. Once you do, a 30-day free trial will start for the core add-in.

2. Install the GSC Connector from within Excel:
Now that the Core Add-in is installed, you need to add the GSC connector so you can work with the Search Console API. It’s very easy to install the connectors available in Analytics Edge. Simply click the License button in the Analytics Edge menu and then click the dropdown to add a new connector. Select Google Search and then click Add. Then click Install. Once the connector is installed, a 30-day trial will begin for that connector.

3. Connect Your Google Account:
In order to export data from your GSC properties, you first need to connect your Google Account associated with those properties. Once you set this up, you will not need to do this over and over. And you can add multiple Google Accounts if you have access to GSC properties across various accounts. Click the Google Search dropdown in the Analytics Edge menu and select Accounts. Click Add Account and walk through the process of quickly connecting your Google account. Once you do, you’ll be able to use the API to connect to any property you have access to.

4. Create Your Macro – Exporting All Landing Pages or Queries and Comparing Data Across Timeframes:
Let’s start creating our macro by exporting all landing pages and comparing data across timeframes. In a real-world situation, you would compare the timeframe after a major event (like a broad core update) to the previous timeframe to see the changes per landing page or query. But for this tutorial, we’ll keep it simple. Let’s just pull all landing pages for the last 28 days and compare to the previous timeframe. Once you get the hang of this, you can customize the report for each situation you encounter. Note, if you ever lose the Macro window, just click “Task Pane” in the Analytics Edge menu in Excel. It will show back up on the right side of the spreadsheet. Let’s start creating our macro. Go ahead and click Analytics Edge in the main menu, then Google Search, and then Search Analytics. Name your macro DeltaReport (or whatever you want).

5. Choose The Account and GSC property:
Select the Google account you want to use and then select the GSC property in the site list.

6. Select Fields To Export:
In the available dimensions and metrics list, select page and click “Add” to export all landing pages from Google organic Web Search. Leave the selected metrics as-is (with clicks, impressions, ctr, and position all selected).

7. Leave Filters Tab As-Is, But Review The Settings:
For this tutorial, we’ll leave the filters tab as-is, but note the options you have here while exporting data from GSC. You can filter by page, query, country, device type, search type, and search appearance. You’ll notice the default search type is Web Search. That’s what we want for this specific report, so keep the default settings.

8. Select A Date Range – Comparing Data Across Timeframes Made Easy:
Depending on your situation, select the appropriate timeframe for exporting data. For this tutorial, let’s pull the last 28 days of data and compare to the previous timeframe. Click the Dates tab and simply use the dropdown to select Last 28 Days. To compare timeframes, make sure to select the checkbox for “Compare to” and select a timeframe to compare with. To keep things simple, we’ll just select “Previous period”.

9. Sort By Clicks Or Impressions:
Under the Sort/Count tab, use the dropdown to select a metric to sort the data by. I typically choose clicks in descending order. Make sure to click the button labeled “Descending” to apply the sort preference.

10. Run The Query!
Click OK in the bottom right corner of the wizard to run the query. Depending on how much data needs to be exported, it can take a few seconds (or longer). Once the query has completed, you will see the results highlighted in green. Note, this does NOT show the full results from the export. Analytics Edge just shows a sample of the results and is waiting for more input from you (either to write the full data to a worksheet now or to use the built-in functions to create more advanced reports).

11. Set The Table Name:
Since we’ll be filtering the data we just exported multiple times (by page type), we need to set the table name so we can come back to the full data in future steps. To do this, click the Table dropdown in the Analytics Edge menu and select Table Name. Set the table name to something like “allpages” and click OK. Again, we’ll need this in the future.

12. Write To Worksheet:
Let’s complete the export by writing the data to a new worksheet. In order to do this, click the File dropdown in the Analytics Edge menu and then select Write to Worksheet. Name your worksheet something like “Landing Pages All Data” and then click OK. Analytics Edge will create a new worksheet containing the full export from GSC. Just click the new worksheet to view all of the data. You’ll notice all of the landing pages were exported with columns showing the difference in clicks, impressions, CTR, and position based on comparing the last 28 days to the previous timeframe. Awesome, right? But we’re not done yet. Our macro will be smarter than that. :)

13. Start Filtering Your Data:
Our goal is to create separate worksheets by page type so you can easily analyze each one separately. To keep things simple, let’s say we wanted to break out category pages (/category/), product pages (/products/), and blog posts (/blog/) so we could analyze them separately. Let’s start with category pages. Click the Table dropdown in the Analytics Edge menu and select “Filter”. This menu will enable you to filter the data by any column in the active table. The active table now is “allpages”, which we set up in step 11. Once you click “Filter in the menu”, you can set the filter rules. The column should say “A page”, which will enable you to filter by the column in our active table titled “page”. For Criteria, you have several helpful options, including regex. Yes, you can use regex here if needed, which is awesome. To filter by category pages which contain /category/ in the url, select “Contains” and then enter /category/ in the Value text box. Then click the Add button. Note, you can combine rules here if you want to create more complex filtering options. Click OK to filter the active table. You will see the results again highlighted in green. We’ll write the filtered data to a worksheet in the next step.

14. Write The Filtered Data To A Worksheet:
Just like we did before, let’s write the filtered category data to a new worksheet. Click the File dropdown in the Analytics Edge menu and select “Write Worksheet”. Then name the new worksheet “Landing Pages Category” and click OK. A new worksheet will be created with all of the category page data. At this point, you should have two worksheets, one containing all landing page data and another just containing category page data.

15. Switch The Table Back In Order To Filter Again:
Now we want to filter the full data again for product pages, which contain /product/ in the url. In order to do that, we need to switch the active table back to “allpages”, which contains all of our exported data and not just the filtered category data. If we don’t switch the table name again, then Analytics Edge will use the current active table, which is the category page data. In order to switch the table, click the Table dropdown in the Analytics Edge menu and click Table Name. Click the second radio button to switch to a previously named table and select “allpages”. Now that becomes the active table and we can filter it again.

16. Filter Product Pages:
We’ll use the same approach that we did when filtering the category pages, but this time, we’ll filter by urls containing /products/. After switching the table name, select Table again in the Analytics Edge menu and then Filter. Now enter /products/ in the value field for page. Then click OK. The data will now be filtered by any url with /products/ in it.

17. Write Filtered Data To Worksheet:
Next, we need to write the product filtered data to a new worksheet. Click the File dropdown in the Analytics Edge menu and select “Write Worksheet”. Name the worksheet “Landing Pages Products” and click OK. You will now have a new worksheet with the filtered data. And now you should have three worksheets containing landing page data (full data, category pages, and product pages).

18. Rinse and Repeat For Blog URLs:
Use the same approach to export all data filtered by blog urls (containing /blog/ in the url). First, switch the table name back to “allpages” (see step 14 for how to do this). Then filter the data by any url containing /blog/, and then write to a new worksheet called “Landing Pages Blog”. When you’re done, you should have four worksheets in total with three that contain filtered data (one for category urls, one for product urls, and one for blog urls). And Analytics Edge already took care of comparing data across timeframes and provided difference columns in the worksheets.

Congratulations! You have exported all of your landing pages from GSC, compared data across timeframes, filtered by page type, and then created specific worksheets containing the filtered data. Oh, and now you have a macro using Analytics Edge that you can reuse whenever you want to accomplish a similar task in the future. Just reopen the spreadsheet, save a new file, edit the macro to change the settings like GSC property, click “Refresh All” in the upper left corner of the Analytics Edge menu, and boom, you’re good to go. Time is valuable and this can save you a lot of it in the future…

Beyond The Delta Report: More Functionality = More Advanced Reporting
As I mentioned earlier, Analytics Edge comes with a ton of functionality built-in. You can create advanced reporting by using the various functions available in Analytics Edge when working with data exported from GSC, Google Analytics, and more. So, if you’re feeling ambitious, here are some other things you can try using Analytics Edge and the GSC Connector:

  • Run the same type of report, but for queries instead of landing pages. Then you can analyze drops or surges by query type instead of page type.
  • You can segment by search type to analyze drops and surges for Image Search, Video Search, or the News tab.
  • You can segment by search feature (like AMP, how-to, FAQs, Q&A, reviews, recipes, etc.) Note, you can follow my tutorial for exporting data by search appearance to learn more about that process.
  • And make sure to review all of the functions available in the Analytics Edge menu within the Multiple, Table, and Column dropdowns. For example, you can filter, group, pivot, sort, append, combine, compare, convert, split, and more. Again, Analytics Edge is like a Swiss Army Knife for APIs.

Summary – Automated Delta Reports are Panda Reports on Steroids
It’s always smart to analyze the top landing pages and/or queries when a site sees a big drop or surge in rankings and traffic from Google (due to an algorithm update, site migration, website redesign, etc.) Since the Performance reporting in the GSC UI limits exports by 1K rows, using a tool like Analytics Edge can help you quickly and efficiently export all of your data via the Search Console API.

In addition, Analytics Edge comes with a number of functions for filtering and working with your data to create advanced reports (including comparing data by timeframe). By following this tutorial, you can create a template for quickly exporting data, comparing data across timeframes, filtering by page or query type, and then writing the results to separate worksheets for further analysis. Once you get the hang of Analytics Edge, the sky’s the limit. I think you’ll dig it.

GG

How To Use GSC’s Crawl Stats Reporting To Analyze and Troubleshoot Site Moves (Domain Name Changes and URL Migrations)

By Leave a Comment

For site migrations, I’ve always said that Murphy’s Law is real. “Anything that can go wrong, will go wrong.” You can prepare like crazy, think you have everything nailed down, only to see a migration go sideways once it launches.

That’s also why I believe that when something does go wrong (which it will), it’s more super-important to address those problems quickly and efficiently. If you can nip migration problems in the bud, you can avoid those problems becoming major issues that impact SEO. That’s why it’s important to prepare as much as you can, have all of the necessary intelligence in front of you while the migration goes live, and then move quickly to attack any problems that arise.

By the way, if you think you’re immune to site migration problems, then listen to the episode of Google’s Search Off The Record podcast where John Mueller, Gary Illyes, Cherry Prommawin, and Martin Splitt talk about the migration of the webmaster central site to the new search central site. It ends up they ran into several problems just like any other site owner could and had to move quickly to rectify those issues. So, if it can happen to Google, it sure can happen to you. :)

Adding Google’s Crawl Stats Reporting To Your Site Migration Checklist:
There are plenty of checklists and tools out there to help with site migrations. For example, Google’s testing tools in GSC, third-party crawlers like Screaming Frog, DeepCrawl, and Sitebulb, site monitoring tools, log file analysis tools, and more.

And on the topic of log files, they provide the quickest way to understand how Google is crawling your site post-migration. You don’t need to wait for data to populate in a tool, you don’t have to guess how Google is treating urls, redirects, etc., and there are several log file analysis tools hungry to consume your logs.

But there’s a catch… trying to get log files is like attempting to complete a mission as Tom Cruise in one of his great Mission Impossible movies. If gaining log files was a scene in Mission Impossible, I could hear Tom now:

“Wait, so we have to scuba dive under a bridge heavily guarded by troops, climb a 200 story building in our underwear (in 20 degree weather), use elaborate yoga moves to dodge a scattered laser security system, steal the ancient lamp of Mueller which is protected by special forces, hack into a computer system protected by six layers of ciphers, download the log files, and then parachute off the building back into the water, only to scuba dive back under the bridge to safety? No problem… hold my coffee.”

OK, it’s not that bad, but any SEO that has attempted to get log files from a client knows how frustrating that situation can be. They are huge files, seemingly not owned by one group or person in a company, and you can even find some companies not keeping logs for more than a few days (if that). So, it’s no easy feat to get a hold of them.

What’s an SEO to do?

Meet The New Crawl Stats Report in GSC: A (Pretty) Good Proxy For Log Files
In November of 2020 Google launched the new Crawl Stats reporting in GSC. The reporting is outstanding, and it was a huge improvement from the previous version. The new reporting provides a boatload of data based on Google crawling your site. I won’t go through all of the reports and data in this post, but you can check out the documentation to learn more about each of the report sections.

I’m going to cover what Google considers “site moves with url changes”, which covers domain name changes and url migrations. I’ll focus on domain name changes, but you can absolutely use the new Crawl Stats reporting to troubleshoot url migrations as well.

For domain name changes, you can view crawl stats reporting for both the domain you are moving to and the domain you are moving from. So, using the crawl stats reporting can supplement your current migration checks and enable you to see how Google is handling the migration at the source (the old domain).

And for url migrations, you can also surface problems that Google is experiencing post-migration. It’s not as clear as a domain name change, since you can’t isolate the crawl stats reporting by domain, but it can still help you surface issues based on bulk-changing urls.

Note: There is a delay in the Crawl Stats reporting.
The Crawl Stats reporting lags by a few days, so log files are still important if you want to see a real-time view of how Google is handling a site migration. The reporting updates daily, but lags by 3-4 days from what I have seen. Below, you can see the report was last updated on 2/26/21, but today is 3/2/21.

How to identify problems with domain name changes and url migrations using the Crawl Stats reporting in GSC:
As mentioned above, for domain name changes, you can analyze the Crawl Stats reporting for the domain name you are moving from, and the domain name you are moving to. Below, I’ll cover some of the ways you can use the reporting to surface potential issues.

How To Find The New Crawl Stats Report in Google Search Console (GSC):
First, I know there’s some confusion about where the new Crawl Stats report is located. You will not find the report in the left-side navigation in GSC. Instead, you first need to click “Settings”, find the Crawling section of the page which contains top-level crawl stats, and then click “Open Report” to view the full Crawl Stats reporting.

Now that you’ve found the Crawl Stats reporting, here are some of the things you can find when analyzing and troubleshooting a site migration.

404s and Broken Redirects:
The Crawl Stats reporting for the domain you are moving from will list urls that Google is crawling that end up as 404s. All urls during a domain name change should map to their equivalent url on the new domain (via 301 redirects). By analyzing the source domain name that’s part of the migration, you can view urls that Googlebot is coming across that end up as 404s. And that can help you find gaps in your 301 redirection plan.

For example, you can see the reporting for a site that went through a domain name change below. 4% of the crawl requests were ending up as 404s when most of the urls should be redirecting to urls that return a 200 header response code on the new domain.

If you click into that report, you can see a sample of the top 1,000 urls with that issue and you can inspect the urls as well:

And here is what it should look like. 100% of the requests are 301 redirecting to the equivalent urls on the new domain:

Important (and often confusing) note: It’s worth noting that GSC reports on the destination url, so a 404 showing up for the old domain name could actually be showing you a redirect to the new domain name, but that new url 404s. In other words, the 404 is actually on the new domain, but shows up in the reporting for the old domain name. That’s extremely important to understand overall with GSC, and it can cause confusion while analyzing the reporting. I tweeted about this in January with regard to the Coverage reporting:

Image Search: Googlebot Image
If image search is important for your business, you will definitely want to review the “By Googlebot type” section of the reporting. You will see a listing for “Image”. You can click into that reporting to see the urls Googlebot Image is crawling. If you see 404s, 5XX, etc., then make sure you jump on those issues quickly. You should see plenty of 301s if you redirected images properly during the migration (which you should). I covered that in my Mythbusting video with Google’s Martin Splitt about site migrations. The video can be seen later in this post.

As you can see below, Googlebot Image is coming across 404s as well. This is from the site that went through a domain name change.

This is what you should see. Notice how the Googlebot Image requests all properly 301 redirect to the images on the new domain:

Robots.txt issues:
In the host issues section, you can see if Googlebot is having problems accessing the robots.txt file for the domain(s) involved. If Google cannot fetch the robots.txt file (which returns a 200 or 403/404/410), then it will not crawl the site at that time. Google will check back later to see if it can fetch the robots.txt file. If it can, then crawling will resume. You can read more details about how this is handled on Google’s support page (or in the screenshot below). Note, you can 404 a robots.txt file and that’s absolutely fine. This is about Google having problems fetching the file (i.e. Google seeing a 429, 5XX).

And here is what it can look like in GSC’s Crawl Stats reporting. Although this falls under an “acceptable fail rate”, I would sure check why the robots.txt fetch is failing at all:

Other host issues: DNS Resolution and Server Connectivity:
Along the same lines, you can see if there are other host-level issues going on. The host reporting also contains DNS resolution errors and server connectivity problems. You obviously want to make sure Google can successfully recognize your hostname and that it can connect to your site.

Performance Problems:
The reporting also will show pages that are timing out for some reason, so keep an eye on that report. You will find that in the “By response” section.

Subdomain Issues:
Hopefully you picked up all subdomains that were in use before you pulled the trigger on the domain name change. But if you didn’t, you can see Crawl Stats reporting per subdomain that Google is crawling. The catch is that you need a domain property set up in GSC for the domain you are moving from (unless you had those subdomains verified and set up already in GSC).

If you did, you could view the crawl stats reporting for those subdomains separately. Domain properties make this easier since all subdomains being crawled by Google (the top 20 over the past 90 days) will be shown in the Hosts report in the Crawl Stats reporting.

Below, you can see that the crawl stats reporting shows 17 different subdomains with crawl requests over the past 90 days.

Note, I always recommend having a domain property set up. It’s amazing how many companies have not done this yet… If you haven’t, I would do that today. It doesn’t take long to set up and it covers all protocols and subdomains.

Crawl Stats For Site Migrations: Final tips And Recommendations
The Crawl Stats reporting can help site owners and SEOs get closer to log file analysis, when gaining those logs might be tough. Although there’s a lag in the data populating (3-4 days), the Crawl Stats reporting can sure help surface problems during domain name changes and url migrations. And the quicker you can nip those problems in the bud, the less chance they become bigger issues SEO-wise.

Here are some final tips and recommendations:

  • Set up domain properties for each of the domains involved in the migration (if changing domain names). This will give you access to all subdomains in the Crawl Stats reporting.
  • Once data starts populating in the Crawl Stats reporting post-migration, dig into the domain you are moving from. You might see a number of issues there based on what I explained earlier. For example, 404s, performance issues, robots.txt problems, and more.
  • Nail the redirection plan. If you see gaps and problems with your 301 redirects, move quickly to rectify those problems. Nip those problems in the bud.
  • Check for host-level problems (like robots.txt fetch issues, DNS resolution issues, and server connectivity problems). Your redirection plan doesn’t matter if Google can’t successfully connect to your site.
  • Look for pages that are timing out. This would show up in the “By response” section of the reporting. If you see that, dig into those problems to see why the pages are timing out. Again, move quickly to address performance issues.
  • Don’t forget your images! Make sure to 301 redirect your images and then check the section labeled “By Googlebot type”. Then check the “Image” reporting to see how Googlebot Image is crawling your content.

More About Site Migrations: Mythbusting Video
If you are interested in site migrations, then you should check out the Mythbusting video I shot with Google’s Martin Splitt. In the video, we cover a number of important topics including domain name changes, url migrations, redirecting images, when a site should revert a migration, site merges, the Change of Address Tool in GSC, and more.

Summary – GSC’s Crawl Stats as a proxy for log files.
After reading this post, I hope you see the power in adding Google’s Crawl Stats reporting to your site migration checklist. The reporting provides a boatload of great information based on Google crawling your site post-migration. I’ve found it extremely helpful while helping companies monitor and troubleshoot domain name changes, url migrations, and more. And remember, Murphy’s Law is real for site migrations. Things will go wrong… which is ok. The important part is how quickly you handle and rectify those problems.

GG

Google Search Console (GSC) reporting for Soft 404s is now more accurate. But where did those Soft 404s go?

By Leave a Comment

On January 11, 2021 Google announced improvements to GSC’s Index Coverage data. With the update, which is clearly apparent in the reporting, Search Console removed the generic “crawl anomaly” issue (and mapped those to more granular issues), changed how urls are reported that are blocked by robots.txt, added a new issue called “indexed without content” as a warning, and then had one final note about soft 404s. That last bullet said, “Soft 404 reporting is now more accurate.” And based on my work with large-scale sites, I was particularly interested in that final bullet.

A number of the sites I help had tens of thousands, or even hundreds of thousands of soft 404s being reported based on the nature of their content. Starting on January 5, 2021, the number of soft 404s reported in GSC fell off a cliff. I’m talking about a severe drop, like these examples:

In order to understand why alarms were going off in my office, and why I’m particularly interested in those drops, let me quickly cover more about soft 404s. Then I’ll explain what I found once I dug into each situation.

A Deeper Look at Soft 404s:
Google explains that a soft 404 is a url that tells the user the page doesn’t exist, but it still returns a 200 header response code (instead of a hard 404). In practice, the soft 404s that Google has surfaced historically have been urls that span different page types. For example, the most obvious is a product that’s no longer available that still returns a 200 response code. It might have very little content and a message telling users the product is not available on the site anymore.

But, other types of pages have showed up in the soft 404s reporting over time. For example, I’ve seen super-thin pages show up there like articles, slideshows, job listings, pages with images or video with little textual content, and more. And on larger-scale sites, those pages can be problematic quality-wise depending on how Google is handling them.

For example, a soft 404 is treated as a hard 404. Google will not index those urls identified as soft 404s. But, if a site had a big soft 404 problem, I often found that Google wasn’t always handling all of those pages types as soft 404s… And that led to many thinner urls getting indexed.

So, if you had 50K super-thin pages and Google saw them as soft 404s, then technically that shouldn’t be a huge problem quality-wise. Since they are seen as hard 404s, Google will obviously not take them into account when evaluating quality.

But, what if suddenly those urls aren’t seen as soft 404s anymore? Well, those urls could be counted when Google evaluates quality overall for the site. That’s a big reason my “quality antennae” went up when I saw tens of thousands, or even hundreds of thousands, of soft 404s suddenly disappear. Are they indexed now, categorized as another issue in GSC and not being indexed, or maybe not even known to Google anymore? I NEEDED TO KNOW!

In addition, soft 404s can impact crawl budget, which can be a concern for larger-scale sites. Note, most sites do not need to worry about crawl budget. Only very large sites (typically sites with over one million urls) need to worry about crawl budget. For example, this site needs to worry about crawl budget:

Google’s own documentation explains that soft 404s can eat up crawl budget. So it’s another concern when it comes to soft 404s (and understanding where they magically went on 1/5/21). In the past, if you saw them flagged in GSC, then you could identify patterns and handle appropriately. But if they suddenly disappear, it could be harder to surface and handle.

In Search Of: Lost Soft 404s
This all leads me back to one question… where did all of those soft 404s go?? After seeing the number of soft 404s drop off a cliff on 1/5/21, I decided to dig into the data across sites and find out what happened. So, I grabbed my virtual pickaxe and headed into the GSC mine. I was determined to find out how those urls were being treated now.

Now, if you choose to dig into search console after a change is rolled out by the GSC product team, you’ll quickly realize that you cannot access historical data. That sucks and means you will have a hard time finding urls previously categorized as soft 404s unless you documented them somehow. I had this covered since I always dig heavily into GSC data when auditing client sites. I was able to open my vault of recent audit documents and resurface many soft 404s that were reported over the past six to twelve months.

Based on digging into the data, I was sometimes relieved with what I discovered for certain situations, but also alarmed by what I found in other situations. I’ll explain more below.

Where Did Soft 404s Go and How Is Google Currently Handling Those URLs?
Below, I’ll cover the various categories of findings based on digging into a number of large-scale sites that previously reported many soft 404s. On 1/5/21, the number of soft 404s dropped off a cliff and I was determined to find out how they were currently being treated.

Super-interesting: “The URL is unknown to Google”
This was incredibly interesting to me. I noticed many url across sites that were once reported as soft 404s now returning “The URL is unknown to Google” when inspected in GSC. It’s almost like Google completely wiped out the history for these urls.

This isn’t a category that shows up in the Coverage reporting for obvious reasons… since it’s NOT known to Google. But, it was known to Google at some point recently. I know that for sure. Google’s documentation does explain more about this categorization, but I’m not sure it answers why the soft 404s are now “unknown to Google”. Here is what Google’s documentation explains:

“If the index coverage status “URL is unknown to Google” appears, it means that Google hasn’t indexed the URL either because it hasn’t seen the URL before, or because it has found it as a properly marked alternate page, but it can’t be crawled.”

I’m going to reach out to Google to learn more about this. I would imagine that Google handling many urls as this category is helping it improve the efficiency of its own systems, but again, I’ll see if Google can comment about this. But if it’s unknown to Google, it won’t hurt your site quality-wise. That’s good.

“Crawled, not indexed” and “Discovered, not indexed”:
I have covered this category a number of times in my other posts about technical SEO and broad core updates. Both “Crawled, not indexed” and “Discovered, not indexed” can signal quality problems and/or crawl budget issues. With “Crawled, not indexed”, Google discovered and crawled the url, but has chosen to not index it. I have often found low-quality or thin content when analyzing this category across sites (or just parts of a site Google isn’t digging for some reason).

So, if your soft 404s end up as “Crawled, not indexed”, then that’s ok in the short-term (since they aren’t being indexed and can’t be held against the site quality-wise), but I would still heavily dig into the urls and handle them appropriately. This is also one of the reasons I believe we need larger exports from GSC’s Coverage reporting! Exporting this category in bulk would help site owners identify more patterns across the site that could be causing problems.

And for “Discovered, not indexed”, Google knows about the url, but has decided to not even crawl it (and clearly not index it). I have seen quality problems and problematic url patterns while analyzing this category of urls. For example, with some large and complex sites with deep quality issues, it’s like Google knows ahead of time what it’s going to find on many of those urls (based on url structure) and doesn’t even want to crawl them.

So, if you see many soft 404s now being categorized as “Discovered, not indexed”, then that’s also ok in the short-term since they can’t be held against the site quality-wise if they aren’t being indexed. But just like with “Crawled, not indexed”, I would dig in and make sure the urls are being handled properly. If they should 404, then make sure they return hard 404s. If they are thin and low-quality, then boost the content there. And if it’s a problematic area of the site, then work on improving that area overall. Remember, “Discovered, not indexed” means Google knows about the urls, but doesn’t believe it’s worth it to even crawl them (let alone index them).

URLs Now Indexed: Danger Will Robinson! Danger!
Yes, this is a super important category. When digging into urls that were previously categorized as soft 404s, I noticed some of them were now being indexed! That can be a dangerous situation for some sites. For example, if they were thin and low-quality urls that Google is now indexing for some reason, those urls will now be taken into account while Google evaluates quality for the site overall. “Quality indexing” is very important (especially for larger-scale sites), so having an influx of low-quality content get indexed is not a good thing.

If you notice many urls that were once flagged as soft 404s now being indexed, then it’s important to handle those urls appropriately now. For example, you can noindex low-quality content, boost the content there (improve it so it meets or exceeds user expectations), 301 it to a newer version of the content, or 404 the pages if they shouldn’t be on the site anymore. But I would not leave them as-is if they are low-quality or thin.

True soft 404s still seen as soft 404s:
Even though many soft 404s disappeared from the reporting in GSC, there were still urls categorized as true soft 404s. For example, pages that were literally blank that returned 200 header response codes, pages that returned “not found” messages on the page, but returned 200 response codes, etc. Again, true soft 404s. I guess this is the “handling soft 404s more accurately” part from Google. For these urls, I would make sure to handle them appropriately. If they should 404, then have them return hard 404s and not 200 header response codes. And if they shouldn’t be seen as soft 404s, then fix or boost the content there.

Moving Forward: What Site Owners Can Do
If you have seen a big drop in soft 404s starting in early January 2021, then I highly recommend digging into the situation. If you don’t have historical data containing urls that were once categorized as soft 404s, there’s not really much you can do to see how those soft 404s are being reported now. But, if you did document soft 404s for your site, then you can inspect those urls to see how Google is handling them now.

Are they “Unknown to Google” now, are they being categorized as “Crawled, not indexed”, “Discovered, not indexed”, are they still soft 404s, or are they being indexed? And if they are being indexed, then they can be counted when Google evaluates quality for the site overall. That’s why it’s important to handle the urls appropriately.

Summary – Know Your Soft 404s
With the latest data improvements in GSC, Google started better handling soft 404s. That’s great, but many sites saw a big drop in urls reported as soft 404s at that time. Since that category in GSC often yielded thin or low-quality content on a site, it’s important to understand how Google is currently handling the urls. Are they still not being indexed, or are they now being indexed and counting towards quality? I recommend digging into the situation for your site and then handling those urls appropriately.

GG

Exit The Black Hole Of Web Story Tracking – How To Track User Progress In Web Stories Via Event Tracking In Google Analytics

By Leave a Comment

How to track user progress in Web Stories via event tracking in Google Analytics.

Google’s Web Stories, previously called AMP Stories, can provide an immersive AMP experience across both desktop and mobile. Google has been pushing them hard recently and stories can rank in Search, Google Images, and in Google Discover. On that front, Google recently rolled out a Web Story carousel in Discover, which can definitely attract a lot of eyeballs in the Discover feed. And those eyeballs can translate into a lot of traffic for publishers.

I’ve covered Web Stories heavily over the past year or so and I’ve written a blog post covering a number of tips for building your own stories. I have also developed several of my own stories covering Google’s Disqus indexing bug and the upcoming Page Experience Signal.

Building those stories by hand was a great way to learn the ins and outs of developing a story, understanding the functionality available to creators, the limitations of stories, and how to best go through the life cycle of developing a story. As I explained in my post covering various tips, it’s definitely a process. Planning, creativity, and some technical know-how go a long way in developing an engaging and powerful Web Story.

From a feedback perspective, analytics can help creators understand how well their story is being received, if users are engaged, and how far they are progressing through a story. Unfortunately, that has been challenging to understand and accomplish for many publishers starting off with Web Stories. And that situation has led me to research a better way to track stories via Google Analytics. That’s what I’ll be covering in this post. By the end, you’ll be tracking Web Stories in a more powerful and granular way. I think you’ll dig it.

Analytics for Web Stories – Confusing For Many Creators
From the start, it seemed like analytics took a back seat for stories. There wasn’t great documentation about how to add analytics tracking and the WordPress plugin originally didn’t even have the option for including tracking codes. That changed recently, which was great to see, but questions still remained about how to best track Web Stories. For example, can you use Google Tag Manager, can you add advanced tracking to understand more about how users are engaging with your story, can you track specific elements in your story, etc.?

Basic page-level tracking in Web Stories.

After looking at basic metrics for my stories in Google Analytics (yawn), I went on a mission to enhance my story tracking setup. Unfortunately, there’s still not a one-stop resource from Google for tracking Web Stories (hint-hint Paul Bakaus), but I was able to dig into various documents and articles and figure out a pretty cool solution that’s easy to set up. I’ll provide that setup below so you can start tracking your own stories in a more powerful and granular way.

Tracking User Progress Through A Web Story: A Simple Goal
If you just add a basic tracking code to your story, you will at least know how many people are viewing the story and gain basic metrics for the page (just like any other page in Google Analytics). But that doesn’t really do Web Stories justice…

Web Stories are a unique combination of pages within a story. In other words, Web Stories string together multiple pages, which make up the larger story. Users can click back and forth to view each page within the story. You can also automatically advance the user to the next page after a specific amount of time. And once a user completes a story, they are presented with a “bookend”, which is a final page that contains information selected by the creator.

With a basic tracking setup, Web Stories are like a black hole. People enter, and you have no idea what’s going on within the story. For example, how many pages have they viewed, how far are users progressing through the story, did they reach the bookend, how long did it take to get to the end, etc.?

Wouldn’t it be awesome to be able to track that information??

The good news is that you can, and it’s pretty easy to set up. Below, I’ll cover how to add the necessary tracking to your Web Stories so you can gain more information about how users are engaging with your stories. And beyond just setting up this level of tracking, I wanted to provide more information about how events and triggers work in stories so you can start testing your own advanced tracking setup. Let’s jump in.

Web Story Tracking: A Top-level View of What We Are Trying To Accomplish
Before I cover the tracking framework you can utilize today in order to better track your Web Stories, let’s cover the basic bullet points of what we are trying to achieve. Here is what we want to achieve:

  • Track user progress through each Web Story you have published. i.e. Track each page within the story to understand how far users are progressing.
  • Document the Web Story title and organize each page within the story so they can be tracked properly.
  • Track when users reach the final page in your Web Story so you can identify how many users actually reach the end.
  • Track when users enter your bookend, which is a special page at the end of your Web Story that contains social sharing and related links. It’s just another way to understand when users have reached the final part of your story.

For example, wouldn’t it be incredible to see the following? That’s a sample Web Story and data for each page within the story. Yep, this is what we want… let’s go get it:

Event reporting in Google Analytics for Web Stories.

The Inner Workings: Events, Triggers, and Variables
Every Web Story issues events as a user progresses through a story. For example, when a user progresses from one page to another within a story, the “story-page-visible” trigger fires every time a new page is loaded. You can capture an event like that and report it in Google Analytics using event tracking.

When sending those events to Google Analytics from within your story, you can provide the typical event parameters like event_action, event_category, and event_label so you can track your Web Story data in your GA reporting.

Event tracking in Google Analytics.

List of Web Story Triggers and Variables:
There are several triggers you can capture and a list of them can found on github for AMP Project/AMP HTML. In addition, you can view the variables available to you via AMP by checking out the AMP HTML Variable Substitutions page. Between the two documents, you can learn how to combine triggers and variables to set up advanced tracking.

Web Story triggers.
AMP variables.

A Template From The AMP Project!
During my research, I was excited to see that the AMP blog published a post about tracking Web Stories and it contained a skeleton structure for advanced tracking! For example, the post listed a code snippet for firing an event every time a user progresses from one page to another within a web story (to track user progress). We could use this snippet, and expand on it, to customize our tracking.

Here is an image from the AMP project’s blog post about tracking user progress. Again, this is exactly what we are looking to do.

Analytics setup for web stories.

By using the right triggers and variables and then firing events from our Web Story, we can get a much stronger picture of user engagement. Below, I’ll provide the triggers and events we’ll use and then I’ll provide the final code later in the post.

Note, there are three triggers we’ll be capturing in our Web Story and we’ll fire an event when those triggers are captured.

  • Trigger: story-page-visible. When each new page in the story loads, story-page-visible fires. When that fires, you will send an event to Google Analytics with the following variables.
  • event_name: You can name this whatever you want. Event tracking in Google Analytics focuses on the following three fields.
  • event_action: Name this something descriptive. For this example, we’ll use “story-progress” which is what the original blog post used covering this approach.
  • event_category: For this field, I’m going to use a variable for Web Stories, which is the title of Web Story. The variable is ${title}, which is what’s present in your title tag. I linked to the variables available to you earlier in this post.
  • event_label: For the final field, we’ll use both the page index value (page number) and ID for the page within the Web Story (which is the descriptive name for the page you provide in your story). This will enable us to see how many times a specific page within the Web Story is loaded by users. The variables are ${storyPageIndex} and ${storyPageID} and you can combine the two in your code. I added “Page: ${storyPageIndex} ID: ${storyPageID}” to combine both in your event reporting. It makes it easier to see the page number and then the ID associated with that page. BTW, thank you to Bernie Torras who pinged me on Twitter about storyPageIndex, which is a great way to capture the page number within your story.

Next, we want to know when users visit the final page of each Web Story. That can help us understand how many people are actually reaching the end. To accomplish that, we can add another trigger:

  • Trigger: story-last-page-visible. Note, this is not the bookend. Instead, this is the last page in your story before the bookend is displayed. Story-last-page-visible fires when a user reaches that final page in your story before the bookend.
  • event_name: You can name this whatever you want. Just like earlier, the reporting in Google Analytics focuses on the following fields.
  • event_action: Name this something descriptive. For this example, we’ll use “story-complete” since the original blog post covering this tracking framework used that action name.
  • event_category: Make sure to use the same event_category for this trigger as you did earlier to keep the various triggers organized by Web Story. The variable is ${title}. Then you can drill into a specific story in Google Analytics and view the actions and labels associated with that one story.  

And finally, let’s add one more trigger to understand when users reach the bookend in your Web Story, which is a special page at the end that contains social sharing and related links. It’s just another way to understand that users made it to the very end of your Web Story. You’ll need to add one more section of code to your tracking script:

  • Trigger: story-bookend-enter
  • event_name: You can name this whatever you want. As I mentioned earlier, the reporting in Google Analytics focuses on the following fields.
  • event_action: You can also name this whatever you want. For this example, let’s use story-bookend-enter.
  • event_category: Like earlier, I’m going to use a variable for Web Stories, which is the title of Web Story. The variable is ${title}. Remember to keep the category consistent with each trigger so you can view all events within a single web story in your reporting.

By adding this setup, the event tracking reporting in Google Analytics will enable you to drill into specific stories, see the number of “pageviews” for each page within a story, know how many users are reaching the final page in a story, and then how many are viewing the story bookend. It’s a much stronger setup than just seeing a single pageview for your Web Story (AKA, the black hole of Web Stories).

Here is the final code based on what I mapped out above. Make sure you replace the placeholder GA account ID with your own:

<amp-analytics type="gtag" data-credentials="include">
  <script type="application/json">
	{
	  "vars": {
		"gtag_id": "UA-XXXXXX-X",
		"config": {
		  "UA-XXXXXX-X": {
			"groups": "default"
		  }
		}
	  },
	  "triggers": {
		"storyProgress": {
		  "on": "story-page-visible",
		  "vars": {
			"event_name": "custom",
			"event_action": "story_progress",
			"event_category": "${title}",
			"event_label": "Page: ${storyPageIndex} ID: ${storyPageId}",
			"send_to": ["UA-XXXXXX-X"]
		  }
		},
		"storyEnd": {
		  "on": "story-last-page-visible",
		  "vars": {
			"event_name": "custom",
			"event_action": "story_complete",
			"event_category": "${title}",
			"event_label": "${totalEngagedTime}",
			"send_to": ["UA-XXXXXX-X"]
		  }
		},
		"storyBookendStart": {
		  "on": "story-bookend-enter",
		  "vars": {
			"event_name": "custom",
			"event_action": "story_bookend_enter",
			"event_category": "${title}",
			"send_to": ["UA-XXXXXX-X"]
		  }
		}
	  }
	}
  </script>
</amp-analytics>

How To Add Your GA Tracking Script To A Web Story:
Once you have your tracking script ready, you need to add that to your Web Story code. I’ve been hand-coding my stories so it’s easy to have control over where the amp analytics tag is placed. In my stories, I place the amp analytics tag after the final page in my story, but before the bookend tag and the closing <amp-story> tag. If you place the amp analytics tag outside of your <amp-story> tag, the Web Story will not be valid. You can see the placement of my amp analytics tag in the screenshot below.

Amp analytics placement in Web Story code.

Make sure your amp-analytics tag is placed before the closing amp-story tag.

Placing amp analytics tag before bookend and closing amp story tag.

A Note About The Web Stories WordPress Plugin:
Again, I have been hand-coding my stories and haven’t dug too much into the WordPress plugin. I’ve heard good things about it, but I really wanted to learn the ins and outs of building a story, so I stuck with hand-coding my stories.

The WordPress plugin finally added the ability to easily include your Google Analytics tracking ID, but it doesn’t look like you can add advanced-level tracking easily (like what I’m mapping out in this post). I’ll reach out to the Web Story team to see if they will add the ability to accomplish this in the future, but for now I think you’ll be limited to the basic tracking I mentioned earlier.

{Update: WordPress Plugin Automatically Firing Events}
I have very good news for you if you are using the Web Stories WordPress plugin. Brodie Clark pinged me today after going through my post. He is using the Web Stories plugin, checked the Events reporting in Google Analytics, and noticed the plugin is automatically firing those events! That’s amazing news for any plugin users!

Again, I’ve been hand-coding my stories so I haven’t played around too much with the plugin. But that’s outstanding news, since plugin users can view user progress and a host of other events being fired within their stories.

Once you add your GA tracking ID, it seems the plugin is automatically capturing the data and firing events:

Adding a Google Analytics tracking ID to the WordPress Web Story plugin.

Here are the triggers being captured based on what Brodie sent me:

WordPress Web Story Plugin automatically firing events.

And here is what it looks like once you click into story_progress. The plugin is using storyPageIndex versus storyPageID so you can see the page number in the reporting. I’m thinking about combining the two actually.

Tracking user progress through Web Stories via the WordPress plugin.

How To Test Your Tracking Via Google Analytics Real-time Reporting
The easiest way to test your new tracking setup is to upload your story to your site and view real-time reporting in Google Analytics. There’s a tab for Events, where you can see all of the events being triggered. Just visit your story and look for the various events, actions, and labels.

Viewing real-time reporting in Google Analytics for Web Story events.

Viewing Web Story Tracking In Google Analytics:
Once your story is live that contains your new tracking setup, and users are going through your story, you can check the Events reporting within the Behavior tab in Google Analytics to view all of the events that have been captured. This is where the naming convention we used comes in handy. When you click “Top Events”, you will see the event categories listed. We used the story title as the category, so you will see a list of all stories where events were captured.

When you click into a story, you will see each action that was captured (each trigger that was captured as an event).

Viewing Web Story triggers in the Events reporting in Google Analytics.

And by clicking an action, you can see the labels associated with that action. For example, story_progress enables you to see the list of pages that users viewed within your story and how many events were triggered for each page (helping you understand how far users are progressing through each story).

Viewing user progress through a Web Story in Google Analytics.

And there you have it! You can now track your Web Stories in a more powerful and granular way. It’s not perfect, but much stronger than the “black hole” approach of just basic page-level metrics. And remember, you can totally expand on this setup by adding more triggers and using the variables available to you.

Summary – Creep out of the black hole of Web Story tracking.
I hope you are excited to add stronger tracking to your Web Stories. As I documented in this post, you can creep out of the black hole of story tracking and analyze user progress through your story. By doing so, you can better understand user engagement and then refine your stories to increase engagement and user happiness.

So don’t settle for black hole reporting. With a basic level of understanding of event tracking, triggers, and variables, you can set up some very interesting tracking scenarios. Good luck.

GG

Image Packs in Google Web Search – A reason you might be seeing high impressions and rankings in GSC but insanely low click-through rate (CTR)

By Leave a Comment

Image pack rankings in Google web search.

While analyzing performance in Google Search Console (GSC), some site owners have reported a strange situation with impressions and position (and some even believe it’s a bug). I’ve seen this many times and I often get questions about it from site owners left scratching their heads after witnessing those metrics in their own stats. Although it’s easy to chalk it up to a GSC bug or anomaly, the answer can be hiding in plain sight.

I’m referring to seeing a site ranking very highly in the search results (according to GSC), but when you check the search results, you can’t find the page ranking. And even stranger, you might see that #1 or #2 ranking with insanely low click-through rates compared to other top rankings (which is often a sign of the situation I’m covering today).

So, what can cause this strange situation in the Google search results with high rankings but insanely low click-through rate? Well, it’s sometimes just an image pack ranking in the SERPs or an image ranking in a knowledge panel. Most site owners scan right past those SERP features and look for high rankings in the 10-blue links, when those very features are often the culprit. Let’s explore this situation in greater detail.

A quick reminder about how Google calculates impressions and position in GSC:
Last year I wrote a post demystifying clicks, impressions, and position in Search Console. In my post, I covered how image blocks were tracked and calculated. For example, if Google was ranking an image block #1 in the SERPs and that block contained 10 images, then each of those images take on the rank of the entire block. Therefore, your image, even if ranking seventh within the image block, would rank #1 in GSC.

If you are surprised by that, which is totally understandable since it’s confusing, then you should definitely read my entire post on Search Engine Land. There are several other surprises in the article that can help you better understand tracking performance metrics in GSC.

Image block ranking in Google search.

Google only counts clicks outside of the search results and not query refinement links. But is that really correct??
To register an impression, Google explains that a link must lead out of the Google search results and to your site. If it’s just a query refinement link (which generates a new search result), then that link will not register an impression.

What is a click in Google Search Console (GSC)?

This is where image packs stand out (and maybe not in a good way).

Image packs contain a series of images from various websites. When clicking a thumbnail, you are not taken directly to the site and outside of the search results. Instead, you are taken to Google Images with that image highlighted and expanded. Then, if you click the image again, you are taken to the third-party site. So the click leading outside of Google doesn’t happen from the web search results. It happens from Google Images.

So, if that’s the case, then images within image packs in the web search results should NOT register an impression. But they do. And that’s super confusing for many site owners, and even experienced SEOs.

For example, here is an image pack ranking in Google for “new york photos”. Notice the image pack ranking at the top of the search results (web search).

Image block for New York Photos in the Google web search results that lead to Google Images.

When clicking an image in the image pack, you are taken to Google Images and not the third-party site that contains the image. But… that image will register an impression and position in GSC. More about that soon (with evidence). After clicking the second image in the image pack, I was taken to Google Images with that image expanded:

Image search results for New York Photos.

The reason that urls from publisher sites do gain an impression is that the image tag’s title attribute contains the url for the site. And that url is the one gaining the impression and position in your GSC reporting. Here is the code showing the title attribute containing the site’s url:

Image pack links actually lead to Google Images but the title attribute contains the url from third-party websites.

To clarify, the image doesn’t link to third-party sites. The link is to Google Images and not to your site. But, the image tag’s title attribute does contain your link, so that is what gains an impression and click in GSC.

Confusing? Yep… And can that throw off your reporting? Affirmative.

But it’s not just image packs. It’s knowledge panels as well:
Just a quick note that this also impacts knowledge panels that contain a block of images. So, if you search for “tesla model 3” and see the knowledge panel containing a number of images, those will behave the same way from a tracking standpoint. They will take on the rank of the knowledge panel (the parent search feature). And to make things even more confusing, a knowledge panel will often rank on the right side of the SERP on desktop, which means it can be in position #11, #12, #13, etc. depending on the query and the resulting search results. The knowledge panel below ranks #11 for this query on desktop:

Knowledge panel in Google search that contains images from publisher sites.

And that same knowledge panel on mobile is at the top of the search results (which would show up as position of #1 in GSC). So you really need to filter your GSC reporting by desktop and mobile to see an accurate position for queries that are impacted.

Knowledge panel on mobile that also contains images from publisher sites.

Now, if you have a lot of images ranking in the SERPs, you might have squeezed your coffee mug so hard it splintered into 35 pieces while reading this post. That’s ok, just clean up the coffee, brew another cup, and get ready for some evidence. Below, I’ll cover several different examples of image packs and knowledge panels registering impressions and clicks in GSC based on simply an image ranking in those features (that actually lead to Google Images and not to the third-party website).

Examples of image pack rankings with GSC metrics:
Below, I’ll provide three examples of image packs or knowledge panels yielding high rankings and low CTR in GSC. I’ll provide the SERP, GSC metrics for web search, and then GSC metrics for image search. I think you’ll get the picture, so to speak. :)

Here’s an image pack ranking for the query “rainbow”. The pack ranks #1 in the SERPs, which means the images take on the rank of the pack (so they also rank #1):

Image pack ranking number one for the query "rainbow".

For image search in GSC, you can see the image has yielded many clicks… but that’s image search. Web search is a different story.

Image search metrics yield stronger numbers.

For web search, there are almost 3M impressions, but only 430 clicks (for a CTR of basically 0%). This is exactly what can happen when an image ranks in an image pack in the web search results:

Web search results for image packs in GSC.

For the second example, here’s an image pack ranking #2 for the query “jaguar suv”.

Image pack ranking in Google for "jaguar suv".

In image search, you can see 562 clicks based on 55K impressions for a CTR of 1%:

Image search statistics for "jaguar suv".

But in web search, there have only been 70 clicks based on 43K impressions. The CTR is only .2%. So although GSC shows strong rankings and 43.8K impressions, the reality is that the ranking leads to almost no traffic at all from Google. Again, this can cause a lot of confusion while site owners are analyzing their performance in Search:

Web search stats in GSC for "jaguar suv".

The last example I’ll provide is from a knowledge panel. There are images ranking in the knowledge panel that make for an interesting case. On desktop, the knowledge panel ranks on the right side (in position 11 for this query):

Knowledge panel with images ranking in the web search results.

But on mobile, the knowledge panel ranks #1 at the top of the SERP:

Knowledge panel ranking in the mobile search results.

The images in the knowledge panel for desktop rank #11 and the images in the knowledge panel on mobile rank #1. The following screenshots from GSC show this is the case. Here are the desktop stats. Yep, just 9 clicks with a position of #11 since the knowledge panel ranks on the right side:

GSC metrics for a knowledge panel ranking in the web search results.

And here are the mobile search stats. 210 clicks with a position of #1 (since the knowledge panel ranks at the top of the SERP on mobile). There were 242K impressions and just 210 clicks:

GSC metrics for a mobile knowledge panel ranking in Google.

And compare both of those to the image search stats… 3.3K clicks for the query. A big difference and shows the gap between images ranking in web search (via image packs or knowledge panels) versus ranking in image search:

Image search results versus knowledge panel results.

Image Packs, Knowledge Panel Images, and GSC: Key Takeaways
As you can see from the examples above, images ranking in image packs and knowledge panels can definitely yield some strange metrics in Google Search Console. It’s important to understand this, since it can lead to a lot of confusion while analyzing a site’s performance in Search.

Below, I’ll cover some key takeaways below based on my own research.

  • First, if you think GSC is wrong, check again, but keep your eyes peeled. There may be an image ranking in an image pack or a knowledge panel that’s throwing off your reporting.
  • I would fully understand how GSC calculates metrics. There are some very interesting and confusing situations based on various SERP features (especially on mobile). I would read my post about calculating clicks, impressions, and position in GSC and Google’s help document covering the subject. Between the two, you can gain a stronger understanding of how everything is calculated.
  • High rankings with extremely low click-through rate could be a sign you are experiencing this issue. If you see high rankings, but very low CTR, that could mean an image is ranking well in web search either within an image pack or a knowledge panel.
  • Although Google explains that only links outside of the search results register an impression, that’s not the case for images in the web search results (within image packs and knowledge panels). The image tag’s title attribute contains the url of the page containing your image, and that’s what is gaining an impression and click.
  • This is a great example of why it’s tough to look at aggregated statistics in GSC. It takes many different variables into account. The devil is in the details (like everything else in SEO). For example, those images ranking well in web search could absolutely be dragging down CTR overall. Make sure you know that while reviewing your stats.
  • It’s also important to understand that Google’s broad core updates can impact Image Search rankings and images that are ranking in the Web Search results (in image packs, knowledge panels, etc.) Google confirmed this to be the case after I wrote a post explaining what I was seeing across sites heavily impacted by core updates.

Summary – Look for image packs and knowledge panels throwing off your reporting:
If you see high rankings with crazy-low click-through rates in GSC and you’re confused about why that’s showing up, you might want to check for images packs or knowledge panels for those queries. You might have images ranking in one of those two search features. It’s a great example of when ranking number one doesn’t really mean very much. Users often scan right by those images in the SERPs, and when that happens, you can end up with a false sense of accomplishment and little traffic to your site. So again, keep your eyes peeled. The answer might be hiding in plain sight.

GG