The Internet Marketing Driver

  • GSQi Home
  • About Glenn Gabe
  • SEO Services
    • Algorithm Update Recovery
    • Technical SEO Audits
    • Website Redesigns and Site Migrations
    • SEO Training
  • Blog
  • Contact GSQi

Night of the Living 302s: How SEO Crawlers and GSC’s Index Coverage Reporting Helped Me Surface A Sinister 302 Redirect Problem [Case Study]

October 31, 2018 By Glenn Gabe Leave a Comment

Share
Tweet
Share
Email
116 Shares

Night of the living 302s.

I recently started helping a new client that has seen a continued drop in search visibility over time. After getting up to speed on the site, its history, business model, etc., I started a thorough crawl analysis and audit of the site. The crawl analysis involves performing an enterprise crawl and then subsequent surgical crawls for areas I want to take a closer look at. I’ve always believed that crawling a site (just like Google does) could help surface potential gremlins that could be hurting a company’s SEO efforts. Unfortunately, what lies beneath from a technical SEO standpoint can sometimes be very scary. Problems invisible to the naked eye can fester over time. Well, this case is a great example of that happening.

Once the first crawls were completed across DeepCrawl, Screaming Frog, and Sitebulb, I quickly noticed something strange. I saw many 302 redirects on the site in the crawl data. It’s not unusual to pick up a number of redirects, but this just seemed higher than usual considering the number of pages present on the site. So I dug in further.

302 redirects *to the homepage*… and many of them
Once I dug into the data, I found thousands of 302 redirects all leading to the homepage. That concerned me on several levels. First, these were real pages with real content that were 302 redirecting to the homepage during the crawls. If you want those pages to rank well over the long-term, then 302 redirecting them is clearly not the route you want to take.

You are basically telling Google that the pages are temporarily being redirected to another page (which was the homepage in this case). And if Google sees 302s in place for a long enough time, then they can treat 302s as 301s. That means they would eventually just index the destination page(s) and drop the pages that are 302 redirecting. Google’s John Mueller has explained that several times in the past.

302 redirects to homepage.

Triangulating The Problem Via Multiple Crawlers
After surfacing the 302 redirect problem, I manually started checking the redirects and noticed the pages were actually resolving with 200s when I checked them via a browser. That was strange… and I started to think that maybe the site was treating the crawlers differently and redirecting them for some reason. So I fired up all three crawls to check the data (across tools) to see if all were seeing the same problem. I wanted to triangulate the data.

Well, all three tools were showing the same thing. The site was indeed redirecting those urls to the homepage during crawling.

Crawlers versus Chrome.

Therefore, manual checks were being handled correctly on the site (the pages were resolving fine), yet crawlers were sometimes being redirected via 302s to the homepage. That was clear across all three crawling tools I was using. Next up, I wanted to check the index coverage reporting in Google Search Console (GSC).

Yep, Index Coverage Also Revealed The Problem
The new index coverage reporting in GSC is awesome. It’s such a huge upgrade from the old index status report and provides a wealth of information directly from Google about crawling and indexing. Although there are several categories in the reporting, nothing comes close to the power of the Excluded category. That’s where you can view a series of reports containing urls that Google is excluding from indexing. And that’s also where you will find redirects. I fired up the reporting, jumped into the Excluded category and found the redirects report.

302s in GSC's index coverage reporting.

Note, although the new index coverage reporting is awesome, there is a slight problem… Unfortunately, you can only view and export the top one thousand listings in any report. That’s typically not even close to enough for larger-scale sites (see the screenshot above). So the Excluded reports contain incredibly actionable data, but it’s somewhat limiting due to the lack of (full) export functionality. Just keep that in mind when you are analyzing the reports.

Note, I know that the GSC team might be working on API access for the index coverage reports. That would be insanely awesome, and incredibly valuable. I hope that’s the case and I’ll be sure to share that news widely if that ever arrives.

Back to the case study. So I checked a number of urls in the report and noticed they did resolve ok when checking them manually. Yet Googlebot was detecting redirects (so Google was running into the same problem that my crawlers were). Then I exported the top one thousand redirects from GSC and started crawling them just to see what my crawlers would return.

Yep, a number of them were 302 redirecting for my crawlers, but resolved with 200s when I manually checked them in a browser. So Googlebot was indeed coming across those 302s just like I was. This was an important find so I immediately crafted a document for my client.

302 redirects in Screaming Frog.

Dev Team Zombie Hunters
After sending a document covering the 302 redirect problem in detail, my client sent that information along to the dev team. After digging into the issue, it ends up there was a load balancing issue that needed to be corrected. So the problem was indeed happening, but it was quietly occurring behind the scenes. It’s yet another example of a sinister SEO problem that can lie beneath the surface. What looks fine when visiting a site might be completely different when Googlebot visits the site. And that can end up very badly from an SEO standpoint.

So luckily, my client was able to push the SEO zombies back into the ground:

SEO Zombies

Based on this case study, and other sinister technical SEO problems I have surfaced in the past, I have provided some quick tips that might help your own situation.

  • Crawl your site just like Google does, and do that on a regular basis. As demonstrated above, you never know what you’re going to find when crawling your site. Everything might be fine, but you might also surface new and serious problems. I recommend crawling your site monthly at a minimum.
  • Always combine crawl data with a manual analysis. Validate what the crawl data is showing by testing the site manually and via multiple devices. Make sure you really have a problem and make sure Googlebot isn’t running into barriers.
  • Triangulate the data via multiple crawling tools. I use three crawling tools extensively when auditing sites. That includes DeepCrawl (where I’m on the customer advisory board), Screaming Frog, and Sitebulb. All three are great and can help you confirm that a certain problem is indeed a problem.
  • Don’t underestimate the power of Google’s index coverage reporting. It’s a powerful set of reports directly from Google covering information about how Googlebot crawls and indexes your site. In addition, the Excluded reporting contains information about urls that Google is excluding from indexing and often contains extremely actionable data. You can also use the inspect url tool to view the latest crawl information, as well as for testing live urls.
  • Implement changes as quickly as you can once you surface problems like this. Technical SEO problems can cause all sorts of crawling, indexing, and ranking issues. The quicker you can rectify those problems, the faster you can recover. And if you haven’t been impacted yet rankings-wise, then you can nip serious SEO problems in the bud.

Summary – Crawlers + GSC Can Help Surface Sinister SEO Problems Invisible To The Naked Eye
Technical SEO is still incredibly important. And what lies beneath could be causing big problems from an SEO standpoint. For example, surfacing problems that Googlebot is encountering, that typical users aren’t, can help you avoid crawling, indexing, and ranking issues. If you’re proactive, and crawl your site on a regular basis, you can help keep the SEO zombies away.

Happy Halloween.

GG

 

Share
Tweet
Share
Email
116 Shares

Filed Under: google, seo, tools

Connect with Glenn Gabe today!

Latest Blog Posts

  • Smart Delta Reports – How To Automate Exporting, Filtering, and Comparing Google Search Data Across Timeframes Via The Search Console API and Analytics Edge
  • Filters and Pills in the Google SERPs – How the addition of filters, tabs, and dynamic organization in the search results can impact visibility and clicks
  • How To Use GSC’s Crawl Stats Reporting To Analyze and Troubleshoot Site Moves (Domain Name Changes and URL Migrations)
  • Google Search Console (GSC) reporting for Soft 404s is now more accurate. But where did those Soft 404s go?
  • Google’s December 2020 Broad Core Algorithm Update Part 2: Three Case Studies That Underscore The Complexity and Nuance of Broad Core Updates
  • Google’s December 2020 Broad Core Algorithm Update: Analysis, Observations, Tremors and Reversals, and More Key Points for Site Owners [Part 1 of 2]
  • Exit The Black Hole Of Web Story Tracking – How To Track User Progress In Web Stories Via Event Tracking In Google Analytics
  • Image Packs in Google Web Search – A reason you might be seeing high impressions and rankings in GSC but insanely low click-through rate (CTR)
  • Google’s “Found on the Web” Mobile SERP Feature – A Knowledge Graph and Carousel Frankenstein That’s Hard To Ignore
  • Image Migrations and Lost Signals – How long before images lose signals after a flawed url migration?

Web Stories

  • Google’s Disqus Indexing Bug
  • Google’s New Page Experience Signal

Archives

  • April 2021
  • March 2021
  • February 2021
  • January 2021
  • December 2020
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • February 2020
  • January 2020
  • December 2019
  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • December 2017
  • November 2017
  • October 2017
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • GSQi Home
  • About Glenn Gabe
  • SEO Services
  • Blog
  • Contact GSQi
Copyright © 2021 G-Squared Interactive LLC. All Rights Reserved. | Privacy Policy

We are using cookies to give you the best experience on our website.

You can find out more about which cookies we are using or switch them off in settings.

The Internet Marketing Driver
Powered by  GDPR Cookie Compliance
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

3rd Party Cookies

This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Keeping this cookie enabled helps us to improve our website.

This site also uses pixels from Facebook, Twitter, and LinkedIn so we publish content that reaches you on those social networks.

Please enable Strictly Necessary Cookies first so that we can save your preferences!