{"id":295,"date":"2013-01-04T10:38:53","date_gmt":"2013-01-04T14:38:53","guid":{"rendered":"http:\/\/www.hmtweb.com\/marketing-blog\/?p=295"},"modified":"2013-01-04T10:38:53","modified_gmt":"2013-01-04T14:38:53","slug":"redirects-duplicate-content-seo","status":"publish","type":"post","link":"https:\/\/www.gsqi.com\/marketing-blog\/redirects-duplicate-content-seo\/","title":{"rendered":"Faulty Redirects, Duplicate Content, and SEO – How a Redirect Glitch Created Hundreds of Thousands of Duplicate Pages"},"content":{"rendered":"

\"Redirect<\/p>\n

With the release of index status in Google Webmaster Tools<\/a>, many webmasters are now questioning why their \u201cnot selected\u201d numbers are high.\u00a0 They wonder if those numbers are good, bad, normal, etc?\u00a0 Unfortunately, there\u2019s not an easy answer to that question, since it depends on the site at hand.\u00a0 But, you can definitely look at the ratio of \u201cnot selected\u201d to pages indexed to start to understand if there is a technical problem causing a spike in pages being categorized as \u201cnot selected\u201d.<\/p>\n

For example, if you have 200 pages indexed on your site, and you see 350 categorized as \u201cnot selected\u201d, that might be ok.\u00a0 But if you see 25K pages as \u201cnot selected\u201d or more, then that could raise a red flag that something may not be right with the site\u2026\u00a0 For example, is there a site structure issue that\u2019s causing thousands of variations of pages with extremely similar content (duplicate content)?<\/p>\n

A Recent Example of a Poor \u201cNot Selected\u201d Ratio
\n<\/strong>During SEO audits, there are times I come across significant problems like the one I mentioned above.\u00a0 And those problems could be inhibiting a company\u2019s search efforts (to say the least).\u00a0\u00a0 During a recent SEO audit, I came across a very interesting situation.\u00a0 Index status revealed an extremely high number of \u201cnot selected\u201d pages (as compared to the number of pages indexed) and I found myself digging into the site to find out why.<\/p>\n

\"High<\/p>\n

I found several issues causing the problem, so there wasn\u2019t just one issue pumping up the number.\u00a0 That said, the problem I\u2019m going to cover today was causing thousands of duplicate pages to be created, and without the site owner knowing.\u00a0 The more pages I checked, the more duplicates I found.\u00a0 And this is a problem that can easily slip through the cracks for many webmasters.\u00a0 And that\u2019s especially the case if a small or medium sized business is handling all website development on its own.\u00a0\u00a0 Below, I\u2019m going to cover what I found, and more importantly, how you can avoid the problem in the first place.<\/p>\n

The Danger of an Extra Character
\n<\/strong>As I was analyzing the site manually, and via a number of test crawls, I came across some URL\u2019s that contained an extra character.\u00a0 Specifically, the extra character was being appended to each canonical URL.\u00a0 All of those URL\u2019s were from one specific section of the site (which contained thousands of URL\u2019s).\u00a0 After digging into that section of the site, I found out that this problem was happening to almost every URL being linked to from a certain element within each page.\u00a0 So, I honed in on that element within each page to find out how the duplicate pages were being created.\u00a0 And by the way, it just so happens that the section of the site contains nearly 200K pages.\u00a0 Yes, this was a huge problem that was uncovered.<\/p>\n

\"How<\/p>\n

The Result \u2013 Duplicate Content Ad Infinitum
\n<\/strong>The core problem is that the extra character created a new URL, but that new URL was an exact duplicate of the canonical URL (the URL that should be resolving).\u00a0 And as you can guess, both pages can be accessed on the site.\u00a0 One part of the site links to the canonical version of the pages, while this problematic section linked to the duplicate versions of the pages.<\/p>\n

So, right off the bat, we are dealing with at least 200K duplicate pages<\/strong>.\u00a0 In addition, as more content is added to this section, more duplicate pages will be created over time (based on the extra character being added to each URL).\u00a0 Also, the canonical URL tag was not being used on the duplicate pages, so that wasn\u2019t helping this specific case.\u00a0 And on that note, I wouldn\u2019t advocate using the canonical URL tag to fix this problem\u2026\u00a0 Technical problems like this should be addressed at the code or structure level.<\/p>\n

\"1<\/p>\n

So, if this was left in place, this problem could generate an unlimited number of duplicate pages.\u00a0 If 500K pages ended up there, then there would be 500K pages of duplicate content.\u00a0 Not good, so I dug deeper to find out exactly what was causing the problem.<\/p>\n

The Root Problem – Faulty Redirects
\n<\/strong>Let\u2019s face it, we all need to implement redirects at some point.\u00a0 And that introduces the possibility of a poor implementation, which can be catastrophic SEO-wise.\u00a0 It\u2019s one of the reasons that website redesigns and CMS migrations are so risky.\u00a0 On that note, to learn how to avoid SEO disaster during a redesign or migration<\/a>, you should check out my Search Engine Journal column on the subject.<\/p>\n

For example, using 302\u2019s versus 301\u2019s, using meta refresh redirects, redirecting to the wrong pages, or having the redirect code bomb the URL\u2019s.\u00a0 For this situation, we ran into the \u201cbombing of URL\u2019s\u201d problem.\u00a0 The redirects were faulty, and were redirecting to URL\u2019s with an extra character.<\/p>\n

The Solution \u2013 Fix the Redirect Code!
\n<\/strong>So, hundreds of thousands of duplicate pages were being generated, and it was due to one piece of redirect code on the server.\u00a0 The 301 redirects being generated simply added an extra character to the destination URL.\u00a0 That\u2019s it.\u00a0 The fix will be implemented soon, and once the new redirect code is rolled out, the correct URL\u2019s will resolve.<\/p>\n

This situation underscores the fact that even one small piece of code<\/a><\/em><\/strong> could have serious implications SEO-wise.\u00a0 If this situation was left unchanged, it could have ended up generating an unlimited number of duplicate pages.\u00a0 Knowing the content on this site, my guess is the problem would have generated 500K-750K pages of duplicate content over the next 2-3 years.<\/p>\n

How To Avoid This Situation
\n<\/strong>After reading this post, you might be scared that this could happen to you, or worse, that it\u2019s happening right now.\u00a0 I\u2019m going to provide a short list of things you can do to make sure this doesn\u2019t happen.\u00a0 Of course, if you feel you are having problems already, you should have an SEO audit performed.<\/p>\n