{"id":2980,"date":"2021-03-02T09:43:52","date_gmt":"2021-03-02T13:43:52","guid":{"rendered":"https:\/\/www.gsqi.com\/marketing-blog\/?p=2980"},"modified":"2021-03-14T07:48:22","modified_gmt":"2021-03-14T11:48:22","slug":"how-to-use-gsc-crawl-stats-report-site-migrations","status":"publish","type":"post","link":"https:\/\/www.gsqi.com\/marketing-blog\/how-to-use-gsc-crawl-stats-report-site-migrations\/","title":{"rendered":"How To Use GSC’s Crawl Stats Reporting To Analyze and Troubleshoot Site Moves (Domain Name Changes and URL Migrations)"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

For site migrations, I\u2019ve always said that Murphy\u2019s Law is real. \u201cAnything that can go wrong, will go wrong.\u201d You can prepare like crazy, think you have everything nailed down, only to see a migration go sideways once it launches.

That\u2019s also why I believe that when something does go wrong (which it will), it\u2019s more super-important to address those problems quickly and efficiently. If you can nip migration problems in the bud, you can avoid those problems becoming major issues that impact SEO. That\u2019s why it\u2019s important to prepare as much as you can, have all of the necessary intelligence in front of you while the migration goes live, and then move quickly to attack any problems that arise.

By the way, if you think you\u2019re immune to site migration problems, then listen to the episode<\/a> of Google\u2019s Search Off The Record podcast where John Mueller, Gary Illyes, Cherry Prommawin, and Martin Splitt talk about the migration of the webmaster central site to the new search central site. It ends up they ran into several problems just like any other site owner could and had to move quickly to rectify those issues. So, if it can happen to Google, it sure can happen to you. :)<\/p>\n\n\n\n

Adding Google\u2019s Crawl Stats Reporting To Your Site Migration Checklist:
<\/strong>There are plenty of checklists and tools out there to help with site migrations. For example, Google\u2019s testing tools in GSC, third-party crawlers like Screaming Frog, DeepCrawl, and Sitebulb, site monitoring tools, log file analysis tools, and more.

And on the topic of log files, they provide the quickest way to understand how Google is crawling your site post-migration. You don\u2019t need to wait for data to populate in a tool, you don\u2019t have to guess<\/strong> how Google is treating urls, redirects, etc., and there are several log file analysis tools hungry to consume your logs.

But there\u2019s a catch\u2026 trying to get log files is like attempting to complete a mission as Tom Cruise in one of his great Mission Impossible movies. If gaining log files was a scene in Mission Impossible, I could hear Tom now:<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

\u201cWait, so we have to scuba dive under a bridge heavily guarded by troops, climb a 200 story building in our underwear (in 20 degree weather), use elaborate yoga moves to dodge a scattered laser security system, steal the ancient lamp of Mueller which is protected by special forces, hack into a computer system protected by six layers of ciphers, download the log files, and then parachute off the building back into the water, only to scuba dive back under the bridge to safety? No problem\u2026 hold my coffee.\u201d

OK, it\u2019s not that bad, but any SEO that has attempted to get log files from a client knows how frustrating that situation can be. They are huge files, seemingly not owned by one group or person in a company, and you can even find some companies not keeping logs for more than a few days (if that). So, it\u2019s no easy feat to get a hold of them.

What\u2019s an SEO to do?<\/p>\n\n\n\n

Meet The New Crawl Stats Report in GSC: A (Pretty) Good Proxy For Log Files
<\/strong>In November of 2020 Google launched the new Crawl Stats reporting<\/a> in GSC. The reporting is outstanding, and it was a huge improvement from the previous version. The new reporting provides a boatload of data based on Google crawling your site. I won\u2019t go through all of the reports and data in this post, but you can check out the documentation<\/a> to learn more about each of the report sections.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

I\u2019m going to cover what Google considers \u201csite moves with url changes\u201d, which covers domain name changes and url migrations. I\u2019ll focus on domain name changes, but you can absolutely use the new Crawl Stats reporting to troubleshoot url migrations as well.<\/p>\n\n\n\n

For domain name changes, you can view crawl stats reporting for both<\/strong> the domain you are moving to and the domain you are moving from. So, using the crawl stats reporting can supplement your current migration checks and enable you to see how Google is handling the migration at the source (the old domain).<\/p>\n\n\n\n

And for url migrations, you can also surface problems that Google is experiencing post-migration. It\u2019s not as clear as a domain name change, since you can’t isolate the crawl stats reporting by domain, but it can still help you surface issues based on bulk-changing urls.<\/p>\n\n\n\n

Note: There is a delay in the Crawl Stats reporting.<\/strong>
The Crawl Stats reporting lags by a few days, so log files are still important if you want to see a real-time view of how Google is handling a site migration. The reporting updates daily, but lags by 3-4 days from what I have seen. Below, you can see the report was last updated on 2\/26\/21, but today is 3\/2\/21.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

How to identify problems with domain name changes and url migrations using the Crawl Stats reporting in GSC:
<\/strong>As mentioned above, for domain name changes, you can analyze the Crawl Stats reporting for the domain name you are moving from<\/strong>, and the domain name you are moving to<\/strong>. Below, I\u2019ll cover some of the ways you can use the reporting to surface potential issues.

How To Find The New Crawl Stats Report in Google Search Console (GSC):<\/strong>
First, I know there’s some confusion about where the new Crawl Stats report is located. You will not <\/strong>find the report in the left-side navigation in GSC. Instead, you first need to click “Settings”, find the Crawling section of the page which contains top-level crawl stats, and then click “Open Report” to view the full Crawl Stats reporting. <\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n

Now that you’ve found the Crawl Stats reporting, here are some of the things you can find when analyzing and troubleshooting a site migration.<\/p>\n\n\n\n

404s and Broken Redirects<\/strong>:
The Crawl Stats reporting for the domain you are moving from will list urls that Google is crawling that end up as 404s. All urls during a domain name change should map to their equivalent url on the new domain (via 301 redirects). By analyzing the source domain name that\u2019s part of the migration, you can view urls that Googlebot is coming across that end up as 404s. And that can help you find gaps in your 301 redirection plan.<\/p>\n\n\n\n

For example, you can see the reporting for a site that went through a domain name change below. 4% of the crawl requests were ending up as 404s when most of the urls should be redirecting to urls that return a 200 header response code on the new domain.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

If you click into that report, you can see a sample of the top 1,000 urls with that issue and you can inspect the urls as well:<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

And here is what it should look like. 100% of the requests are 301 redirecting to the equivalent urls on the new domain:<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

Important (and often confusing) note:<\/strong> It\u2019s worth noting that GSC reports on the destination url<\/strong>, so a 404 showing up for the old domain name could actually be showing you a redirect to the new domain name, but that new url 404s. In other words, the 404 is actually on the new domain<\/strong>, but shows up in the reporting for the old domain name. That\u2019s extremely important to understand overall with GSC, and it can cause confusion while analyzing the reporting. I tweeted about this in January with regard to the Coverage reporting:<\/p>\n\n\n\n

\n

Reminder: GSC reports on destination urls in the Coverage reporting. So if you see urls that are categorized as blocked by robots.txt or noindexed, but they aren't, they could be redirecting to urls that are. And that's what is reported. Can send you off on a wild goose chase: pic.twitter.com\/QYSWUcVTc1<\/a><\/p>— Glenn Gabe (@glenngabe) January 5, 2021<\/a><\/blockquote>