The Internet Marketing Driver

  • GSQi Home
  • About Glenn Gabe
  • SEO Services
    • Algorithm Update Recovery
    • Technical SEO Audits
    • Website Redesigns and Site Migrations
    • SEO Training
  • Blog
    • Web Stories
  • Contact GSQi

Gather, Crawl, Freak, and Fix – How To Run A Quick Health Check Of Top Landing Pages When Seeing A Drop In Organic Search Traffic

January 6, 2016 By Glenn Gabe 4 Comments

Share
Tweet
Share
Email

How To Run A Quick SEO Health Check

It can be extremely frustrating when experiencing a slow decline in rankings and organic search traffic. Could it be an algorithm update, technical SEO problems, or something else? The fact of the matter is that it could be a number of serious problems causing the drop. And unless you can isolate the problem, it’s hard to rectify. And when there’s confusion, it’s easy to implement changes that can either have no impact at all, or that make the situation worse. It’s not good, to say the least.

The Call About A Decline in Google Traffic + A Quick Health Check
After my last Search Engine Land column about robots.txt problems and a slow decline in organic search traffic, I had a company reach out to me about a similar decline. They just couldn’t understand what was going on. Also, they explained that they were not SEO-savvy, they had made some site changes over the past twelve months, and simply had no idea why they dropped in rankings.

As you can guess by the bolding above, I heard “site changes” and decided to do a quick health check of their top landing pages from organic search. Similar to the case study I wrote on Search Engine Land, it’s important to ensure the top landing pages from organic search still resolve properly, aren’t experiencing technical problems, content-related problems, etc.

But First, A Clean Search History (Algo-wise)
One of the first things I do when helping a new client is perform a search history analysis. I wrote an entire post about it on Search Engine Land and I highly recommend you check it out. In a nutshell, I match Google organic search trending with both confirmed and unconfirmed algorithm updates. It helps all involved better understand the impact over time, and which algorithms could be causing problems. It’s a relatively quick deliverable, but sets the stage for a deep audit and analysis.

Example of search trending over time

I was specifically looking to see if the drop in traffic lined up with any algorithm updates from 2015. It ends up the search history analysis did not reveal any crazy hits during the various algorithm updates I have tracked. So the decline wasn’t from Panda, Phantom 2, the mobile-friendly update, or any of the quality updates that rolled out in 2015. That finding, along with the “site changes” quote from earlier led me to think there could be some technical SEO problems on the site. My next step was to perform a quick health check, as noted earlier in the post.

Crawling Top Landing Pages From Organic Search – Bingo
So I quickly jumped into Google Search Console, Google Analytics, and Bing Webmaster Tools to check out the top landing pages from organic search (from before the drop started). Upon checking a few dozen manually, I noticed some strange things. Some resolved fine, while others had various issues. There were some 404s, 500s, some title and content issues, a few weird redirects, and other misc. problems. And that’s just checking thirty to forty urls.

Top Landing Page 404 Header Response Code

It was time for a crawl. There were about 12K deduped urls based on pulling landing page data from GSC, GA, and BWT. I scheduled two crawls of the urls and went back to manually checking urls. I had a feeling the crawl would yield some answers by checking all the urls in bulk.

When A Health Check and Quick Crawl Yield Pure Gold
Once the crawl completed, it was clear why the site was experiencing a drop in organic search traffic. Many of the pages that were once ranking were not resolving properly. Just like I found during the spot checks I performed earlier, the same problems were showing up in bulk now.

For example, I surfaced 404s, 500s, pages literally without content, weird redirects to unrelated pages, etc. And when pages don’t resolve properly, they have little shot of maintaining strong rankings over time. That led to the decline that the company was experiencing.

Screaming Frog Crawl Response Codes

I sent my findings through and my client is working with their technical team to better understand how this happened. Some of the problems look like they were human error, while other problems were related to their CMS. The combination led to many pages not resolving properly. And that led to a drop in rankings and organic search traffic. It’s a great example of how a search history analysis, a health check, and a quick crawl of top landing pages could yield important findings SEO-wise.

I don’t know if the company can recover all of the lost rankings and traffic at this point, but uncovering the core problem is a great start. They now have a smoking gun (which you don’t always have SEO-wise). Let’s face it, it’s important to know what’s riddling the site in order to rectify those problems. And only then can you begin to recover rankings and traffic.

Now It’s Your Turn To Gather, Crawl, Freak and Fix
Some of you might be reading this post and finding the story about a decline in traffic over time extremely familiar. If that’s the case, then I highly recommend following the approach I took with this case and running a quick health check of top landing pages. Those urls may be fine on your site, but they might not be. It’s always smart to run a scan like this from time to time (especially when you see a drop in traffic).

I recommend starting with a search history analysis, identify any drops based on algorithm updates, and document your findings. After which, you should gather and crawl your top landing pages from organic search (including before the drop started). I have provided a bulleted list below of the steps involved.

How To Run A Health Check of Top Landing Pages:

1. Gather Your Intel
Head over to Google Analytics and export all landing pages from organic search. I wouldn’t keep this isolated to Google organic, but you can if you want. If you want all landing pages from organic search (without sampling), then access the Channels report, then Organic Search, and then dimension by landing page.

Top Organic Search Landing Pages in Google Analytics

2. Fire Up Google Search Console (GSC) and Export Top Landing Pages
Note, you can only export the top one 1,000 urls for the site at hand via the UI. But that’s not entirely accurate. If you use filtering with the Pages grouping report, you can isolate specific urls or directories on your site. When you export that filtered report, it will provide that filtered data. So, you can get much more than just 1,000 urls, but it involves some manual work.

Top Pages in Google Search Console (GSC)

3. Don’t Forget About Bing Webmaster Tools
Yes, there is such a thing! And you can export all top landing pages that are receiving impressions and clicks from Bing. You can set the timeframe for any dates during the past six months (which is more than GSC gives you with just three months.)

Top Pages in Bing Webmaster Tools (BWT)

4. Choose Your Crawler of Choice
I use both DeepCrawl and Screaming Frog extensively. DeepCrawl is an amazing enterprise-level crawler and can handle large sites and many urls very well. Note, I’m on the customer advisory board for DeepCrawl. I also use Screaming Frog for smaller websites and for surgical crawls. It’s common for me to use both tools when analyzing a website. Each tool enables you to upload a list of urls to crawl. And that’s what you want to do here after gathering your intel. Then crawl away…

Uploading a list of URLs to DeepCrawl

5. Analyze The Crawl
Once the crawl completes, you should dig into the data to understand how those urls are resolving. There are many things you can find, so I can’t cover them all here. That said, below is a list of some important things you can surface. You might be surprised by what you are seeing (which can be good and bad).

  • 404s
  • 500s
  • Faulty Redirects
  • Noindexed Pages
  • Pages Noindexed Via the X-Robots Tag (included in the header response)
  • Titles and Metadata Changed or Removed
  • Primary Content Removed or Broken (resulting in extremely thin content or user engagement problems)
  • Robots.txt Blocking URLs From Being Crawled
  • Canonicalization Problems
  • And more…


Bonus: Think you’re done? Not so fast. Go Mobile…
Remember, we live in a mobile world. And Google has a crawler focused on smartphones called Googlebot for Smartphones. So I highly recommend running a second crawl that uses Googlebot for Smartphones as the user agent. You might find that your site is handling mobile visitors differently than you thought… For example, you could find faulty redirects, 500s, content problems, etc.

Crawling as Googlebot for Smartphones in DeepCrawl

Summary – You Won’t Know Until You Pull The Trigger
It’s smart to run a health check periodically to ensure your top landing pages from organic search are resolving ok. And I would not wait until you see a decline in traffic to go through the process listed above. I would try to check your urls monthly or bi-monthly at least. By doing so, you can pick up potential problems before they cause a drop in rankings and traffic. The good news is that you can start today. Go ahead and gather your intel and run a health check. You never know what you are going to find. Crawl away.

GG

 

Share
Tweet
Share
Email

Filed Under: google, seo, tools

Connect with Glenn Gabe today!

Latest Blog Posts

  • How to compare hourly sessions in Google Analytics 4 to track the impact from major Google algorithm updates (like broad core updates)
  • It’s all in the (site) name: 9 tips for troubleshooting why your site name isn’t showing up properly in the Google search results
  • Google Explore – The sneaky mobile content feed that’s displacing rankings in mobile search and could be eating clicks and impressions
  • Bing Chat in the Edge Sidebar – An AI companion that can summarize articles, provide additional information, and even generate new content as you browse the web
  • The Google “Code Red” That Triggered Thousands of “Code Reds” at Publishers: Bard, Bing Chat, And The Potential Impact of AI in the Search Results
  • Continuous Scroll And The GSC Void: Did The Launch Of Continuous Scroll In Google’s Desktop Search Results Impact Impressions And Clicks? [Study]
  • How to analyze the impact of continuous scroll in Google’s desktop search results using Analytics Edge and the GSC API
  • Percent Human: A list of tools for detecting lower-quality AI content
  • True Destination – Demystifying the confusing, but often accurate, true destination url for redirects in Google Search Console’s coverage reporting
  • Google’s September 2022 Broad Core Product Reviews Update (BCPRU) – The complexity and confusion when major algorithm updates overlap

Web Stories

  • Google’s December 2021 Product Reviews Update – Key Findings
  • Google’s April 2021 Product Reviews Update – Key Points For Site Owners and Affiliate Marketers
  • Google’s New Page Experience Signal
  • Google’s Disqus Indexing Bug
  • Learn more about Web Stories developed by Glenn Gabe

Archives

  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • August 2021
  • July 2021
  • June 2021
  • April 2021
  • March 2021
  • February 2021
  • January 2021
  • December 2020
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • February 2020
  • January 2020
  • December 2019
  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • December 2017
  • November 2017
  • October 2017
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • GSQi Home
  • About Glenn Gabe
  • SEO Services
  • Blog
  • Contact GSQi
Copyright © 2023 G-Squared Interactive LLC. All Rights Reserved. | Privacy Policy
This website uses cookies to improve your experience. Are you ok with the site using cookies? You can opt-out at a later time if you wish. Cookie settings ACCEPT
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience. You can read our privacy policy for more information.
Cookie Consent