Over the past few years, Google’s John Mueller has been asked many questions about where to best redirect pages during a redesign or website migration. It’s a hot topic, but that makes sense. When you are changing urls, it’s extremely important to redirect older pages to the newer urls using a one to one mapping, if possible (via 301 redirects). For example, product A should redirect to the new url for product A on the new site.
But what about situations where you won’t have a corresponding page on the new site? For example, maybe the product or service isn’t available anymore, a category has been removed, etc. And maybe those pages held a lot of search equity. They rank well, have built inbound links, etc.
When running into situations like this, many webmasters are confused with what to do. Should you redirect the older pages to a different product or category? What about redirecting to the homepage? I mean, you want to save as much search equity as possible, right? But will that actually work??
John Mueller’s Response and Soft 404s
Back to my first point about questions that John has been asked and his response. He has always said that redirects to the homepage or non-relevant pages can be treated as soft 404s. That means Google will essentially view the original pages that are being redirected as 404s. And that means Google will eventually remove those pages from the index (treating them as standard 404s), even though they properly redirect to other pages on your site.
Here’s a quick diagram representing the situation:
So, when this happens, you won’t be retaining search equity during the move like you thought, keeping those strong rankings, and subsequent traffic from those pages. Yes, read that last line again. Signals will apparently not be passed to the destination page via the 301 redirect.
So your nifty redirect to save search equity will actually mean nothing. The new page will not retain the rankings and traffic of the old page (as Google is seeing the page as a 404, and Google will ultimately remove that page from its index). That’s an incredibly important point to understand.
Here’s a video clip of John Mueller explaining the situation in a webmaster hangout video:
In addition to the clip above, John Mueller commented once again today in another webmaster hangout video. Barry Schwartz covered that on Search Engine Roundtable this moring. Crazy timing!
From a user perspective, this makes complete sense. If someone searches for a specific category, product, or service, and you redirect them to the homepage of your site, or a less-relevant category page, then that’s not the greatest user experience. Actually, it can be extremely frustrating for users.
The Summer of Migrations, Again
Last year I declared July and August “the summer of migrations” for me. I had several clients pulling the trigger and migrating to a new CMS. I didn’t think I would match that number of migrations again. Well, I was wrong. The summer of 2016 beat that number already, and we’re only half way through the summer… So here I am again, in the Summer of Migrations Part 2. :)
Several of the migrations this summer involved large-scale websites. And when you have large-scale websites, the size and complexity can throw several wrenches into your operation (including some gremlins). To make a long story short, several clients had categories, services, or product pages that would not be present on their new sites (in the new CMS). But those pages ranked well, had built links, etc.
Several of the companies were moving at light speed to migrate to a new CMS, they were dealing with a lot of content (tens of thousands of urls or more in some situations), and some clients had multiple migrations going on at the same time (more than one site being migrated to a new CMS).
Based on their hectic schedule and the sheer amount of content that needed to be migrated, some clients made the decision to simply 301 redirect older pages with no equivalent page on the new site to top-level categories, other less-relevant categories, and in some cases even the homepage.
What happened next was ultra-interesting to see and confirms what John has been saying for a long time now.
Soft 404 Warnings in GSC
It wasn’t long after the migrations went live before the first warnings started showing up. GSC was sending warnings for a spike in soft 404s based on each migration that implemented redirects to less relevant pages.
And with Google treating those pages as 404s, rankings and traffic began to drop quickly for the older pages that were now being redirected to less relevant pages. So the morale of the story is that you won’t be tricking Google anytime soon with 301 redirects to less-relevant pages. This is important to know for any website that will be changing urls, going through a redesign, or CMS migration.
If you want to retain strong rankings and traffic from Google, make sure you have a one to one mapping from the old url to the new url. If you don’t, and Google sees the new destination pages as less-relevant (or non-relevant), then they could end up as soft 404s. And if that happens, say goodbye to your strong rankings and traffic.
But again, this makes sense for users. If someone is searching for “x category” and you redirect them to “z category”, which isn’t really the best content based on their query, then can you blame Google for treating the old page as a soft 404? I would find that user experience very frustrating, and many others would too.
How To Handle Redirects When URLs Change:
The best solution, and the one I mentioned earlier, is to use a one to one mapping for any important older urls on the site. For example, if you have an important category page that’s ranking well, driving traffic, has built powerful links, etc., then you should absolutely redirect that url to the equivalent url on the new site (via a 301 redirect).
Don’t simply 301 to a less relevant category, or to the homepage. If you do, there’s a good chance Google will understand that and treat the urls as soft 404s. And then you’ll lose rankings and traffic anyway!
If for some reason you don’t have a one to one mapping, and you can’t create a new page closely tied to the older content, then just 404 the page. It’s completely natural to have 404s from time to time. When you 404 the old page, Google will remove the url from its index, users will see your custom 404 page, and you won’t falsely believe that you can retain the signals from the old page.
As John explained in the webmaster hangout video, just make sure your custom 404 provides enough value for users. For example, providing helpful information and links based on the page that has been removed. Basically help them find more, and better, information on your site!
An example of a helpful 404 page:
But to be clear, 404s should be a last-case scenario, especially with pages that are ranking well, driving traffic, and have built powerful links. If possible, make sure all of your top landing pages have a home on the new site. Then your strong rankings can remain, traffic from organic search and referring sites will remain, search equity will remain (based on links pointing to that content), etc. It’s only in situations where you absolutely cannot have an equivalent page on the new site that you should 404 the page.
Note, to download your top landing pages you can follow my tutorials for using Analytics Edge to seamlessly download those urls from Google Analytics and Google Search Console. Using Analytics Edge, you won’t run into the 5K limit per download in GA or the 1K download limit in GSC. So you can download ALL of your landing pages. :)
Summary – You Probably Won’t Trick Google With 301 Redirects To Less Relevant Pages
To wrap up, Googlebot is pretty smart when it comes to redirects to less relevant pages. John Mueller has explained several times that redirects to less relevant pages (or the homepage) can be treated as soft 404s. That means Google will essentially treat those pages as traditional 404s and eventually remove them from the index.
So make sure that you account for your top landing pages correctly when changing urls, migrating to a new CMS, or redesigning your site. There’s a good chance you won’t trick Google with a 301 redirect to less relevant pages. You might just receive a fresh message from GSC about soft 404s. :)
GG
Thanks for the interesting case study, Glenn.
Another point: Google may forget or remove your older pages from the index/archives after awhile and may automatically soft 404 them even if you redirect to a relevant page. Google may not be able to know that the redirect is to a relevant page, because the old variant has long gone from the index.
Hunting 404s from migrations that happened many years ago and redirecting them may also be treated as soft 404s. In such situations, it might be better to find the pages that linked to the older variants and ask the owners to update the link to the new URL as introducing a redirect at such a late stage may just lead to a soft 404.
In other words, migrations need to be done carefully as it might be too late to fix link equity issues at some point later.
Yes, great point Hristo. I actually wrote an entire post about that a few months ago. That post was based on comments from Gary Illyes regarding the loss of signals for pages that were removed from the index for an extended period of time. Actually, I see you commented on that post too! So thanks for bringing that up again. It’s an important point to understand. You can read that post here -> https://www.gsqi.com/marketing-blog/can-you-recover-lost-search-rankings-after-botched-website-redesign-or-cms-migration/
Basically, if the pages 404 and are removed from the index for just a few days, and then they are added back, you will probably be ok. But if they remain out of the index for weeks or more, then the urls may never regain some signals. And that makes complete sense based on helping companies that call me with botched CMS migrations or redesigns.
Thanks Glen Gabe for the clarity. This underscores the need for immediacy. Both points are appreciated: 1. It is cleanest to update the older variants with the link to the new URL.2. And to stay on top of 301 redirect issues with daily monitoring.
It sounds like things have gotten dangerous for any kind of 301. As an example, if you buy an existing website or decide to merge two existing websites the redirects from one to the other might be worthless if you simply redirect them to the homepage. It might even be dangerous to redirect them to a same-topic page because you don’t know if google is going to consider them to be equivalent. I am betting the buying an expired domain and redirecting it is almost worthless today.
I wouldn’t call it dangerous. You won’t get penalized for redirecting to less relevant (or non-relevant) urls. But you won’t really gain anything by doing so…Business owners just need to know how the redirects will work (or not work).
Also, there are nuances with migrating one domain to another. For example, you would submit a change of address in GSC to tell Google that you are moving X domain to Z domain, and you would typically be moving that content to the primary domain. Just buying one domain without moving that content over will not yield the results you would be looking for SEO-wise. The point is to migrate the content from X domain to Z domain and then redirect all the urls (or a majority of them) to the new domain. I hope that helps (I can write an entire post on this topic!) :)
Hi Glenn.
A bit late the the party here but do you think there’s a chance that Google would penalise you for having too many soft 404’s that you created with inaccurate 301 redirect mappings? i.e site rankings suppressed in Panda/Quality filters?
I haven’t got any evidence to this effect but it seems to me that during the time that old urls are still indexed the poor search experience that results via the redirects is precisely the kind of thing that pisses Google off….
Hi Chris. No, I don’t think that’s the case. I think the true damage comes from the pages being dropped from the index, and then rankings and traffic follow. Any rankings you had for the pages that are now redirecting will be lost. And that can be extremely problematic if widespread (which I have seen with some migrations). I hope that helps.
So, basically, if you don’t have a page to redirect the old content, i think, the best thing to do is create a page with a similar content. Maybe you could stay with the relevance and links of the old content. But, obviously, you need to analyse the relevance of the old page to make sure if this work is worthwhile.
Yes, a one to one mapping is best. And if you don’t have that, you can absolutely create content that is equivalent to what was on the old page. That’s definitely an option. If the page is important, ranking well, has built powerful links, etc., then that’s a strong way to go. Basically, keep it if you can (or build it). If you can’t and you need to move on, it’s totally ok to 404 it.
Or, as AJ listed above, you can keep the redirect in place with dynamic messaging (but just don’t think it will help SEO-wise). It can be helpful for users, but the redirects will not pass signals to the new urls.
I think it’s safe to say that the best option is always to 301 in a horizontal fashion, meaning that the content has to be on the same ‘node’ (product A to product B) and not vertical (product A to category A).
That being said there’s a bunch of nuance to this in my opinion. Sometimes there are other reasons to choose a 301 versus a 404. Maybe there are a bunch of pages that are just truly dead and Google keeps crawling them. For whatever reason they often keep getting pinged no matter how many times you play whack-a-mole with them in GSC.
So a redirect can sometimes shorten the timeframe involved and get you to a clean crawl far quicker. At that point I don’t care about link equity. I care about crawl efficiency. So does my ‘illegal’ 301 help me get there faster? I can’t say for sure but … it seems that way.
I’d also say that if you do the redirect but do so with JS messaging then the user experience actually improves. So if product A now redirects to category A with a message that product A was discontinued but that they have other similar items for them to browse … well, isn’t that a good thing? Haven’t I reduced the friction for that user and landed them on the best alternative?
Even the best 404 page is likely going to chase more people away than an explanation and engaging experience right at their fingertips.
If I go to my favorite froyo store they might have a sign up that says the dutch chocolate is no longer available but the kona coffee flavor has now arrived. That’s a whole lot better than having an empty froyo tap that tells me I can find other flavors elsewhere in the store.
That’s not the best analogy but … it’ll do for now. Good stuff here Glenn.
Great points AJ. I think the core point of my post was to make sure business owners understand that 301s to less relevant (or non-relevant) pages can be treated as soft 404s (and will not pass signals). But that doesn’t always mean you have to 404 the pages. You can keep the redirects, but they just won’t help SEO-wise. And I like your idea of dynamic messaging for users based on the redirect. Strange redirects to somewhat related pages can be alarming for users. So the messaging (as long as it’s apparent), can really help there.
What if we migrated a site 3 years ago. Is it too late to go back and re-assess the 301 redirects at this point?
Great question and relates to another one of my recent posts. You can go back and properly redirect older urls to the right ones on the new site, but just understand that not all signals will be passed (due to the length of time). Gary Illyes explained this in a virtual keynote a few months ago. He explained that if a page is removed from the index for a few weeks or more, some signals will be lost. I’ve seen this many times after companies contact me about botched migrations. Important urls end up 404ing, the company doesn’t fix that for a few months, then they do, but rankings do not return (or at least not to the level prior to the migration).
So it’s smart to do to regain some signals, referral traffic, etc. and it also gives you the opportunity to build rankings again for that content. But, it won’t be the same as properly redirecting the urls from the start of the migration. Here’s a link to my post about the topic -> https://www.gsqi.com/marketing-blog/can-you-recover-lost-search-rankings-after-botched-website-redesign-or-cms-migration/
This is important news – thanks Glenn —> don’t slack off when 301ing ;-)
Thanks Andy! And yes, it’s extremely important to understand how 301s to less-relevant pages work, especially when going through a large-scale redesign or CMS migration. :)
It may also explain why redirecting an expired domain to an unrelated fresh domain is inefficient.
Excellent point. And then add the danger of not knowing the full history of that domain (for those that don’t fully research it), and you can see why it’s just not worth the effort.
I wouldn’t redirect a domain for SEO purposes. Google can just treat those redirects as soft 404s unless it’s truly a one-to-one mapping content-wise. Also, you definitely need to heavily research the history of that domain. You don’t want to push algorithmic demotions or manual actions to your current domain. I hope that helps.
So you you are saying just let the old pages not being moved become a 404 error. Is there any value to adding the pages that are not being migrated to a robots.txt file as disallow? I am not moving to a new domain.
So you are not migrating some content? If there are urls not being moved to the new site, then 404ing is fine. If the urls should be removed from the site for whatever reason, then a 404 is totally ok (and normal).
But if you want to retain rankings and traffic for queries leading to those urls, I would either migrate that content or 301 redirect it to an equivalent url (maybe an updated version of the content). That’s if you can do that… Just don’t 301 redirect to a non-relevant url. If so, you can see that Google will often treat those as soft 404s.
You can also choose to boost older and outdated content at that point and then you will be able to migrate the urls. I hope that helps.
OK thanks, this clears things up. I will make sure they have a good 404 page then.