tools | The Internet Marketing Driver

Measuring Infinity – How to identify, analyze, and fix an infinite spaces problem using a powerful SEO software stack

February 8, 2018 By Glenn Gabe Leave a Comment

Infinite spaces and SEO.

If you are working on a large-scale site, you need to be very careful that you don’t create infinite spaces. That’s when a site generates a near-endless list of urls that Google can end up crawling. Google does not want to churn through endless urls and it can cause problems on several levels SEO-wise. That’s why they detail infinite spaces in the search console help center and they also wrote a blog post specifically about the problem.

Google can end up wasting resources crawling many urls that are near-identical or low-quality. And that can flood Google’s index with thin and low-quality content.

During a recent audit on a site that has seen massive volatility over the past few years during major algorithm updates, I noticed a certain page type that I thought could be problematic. This was surfaced after the initial crawl (which was just a few hundred thousand urls out of 30+ million indexed). I saw enough of the problematic page type that I flagged it and then decided to dig in further.

The Core Problem: Autogenerating URLs
The core problem with the page type was that the site was autogenerating urls based on site searches, which ended up adding many thin or near-empty pages to the site (that were indexable). And even the urls that did contain some content didn’t really make sense to have on the site since they were so general (and even ridiculous in some cases). This created a near-infinite spaces situation with millions of urls being generated, and many of them with major quality problems.

I sent through initial findings to my client and we decided to get on a call to discuss the issue, along with a number of other important findings from my initial analysis. After speaking with my client and his tech lead, I found out the problem could be impacting up to 10 million urls. And with the site seeing major volatility during multiple major algorithms updates over the past several years, I couldn’t help but think the infinite spaces problem could be heavily contributing to that volatility.

To Infinity and… Behind? The Plan of Attack:
We quickly decided that I should dig into the situation full-blast to learn more about the autogenerated url problem. I literally left the call and started analyzing the pages using a number of tools. This was a process I’ve used a number of times before, so I thought it would be helpful to write a post covering it. It’s a strong way to use a powerful SEO software stack to surface and analyze an infinite spaces problem.

I’ll provide the process below, along with the SEO tools I used (which include both free and paid tools). The combination enabled me to tell a powerful story quickly and efficiently. My hope is that you can use this process, as well, in case you run across a problem that’s creating infinite spaces.

1) Starting with an enterprise crawl via DeepCrawl and Screaming Frog
The first step was running an enterprise crawl. The site has over 30 million pages indexed, but I almost never start by crawling millions of pages at once. You can definitely get a good feel for a site by crawling a few hundred thousand pages to start, and then perform surgical crawls based on problematic areas you surface. It’s a quicker and smarter way to crawl when you’re just starting an engagement.

The initial enterprise crawl helped surface many thin or near-empty pages, and those urls were the problematic page type I mentioned earlier. When digging into those urls, you could clearly see the infinite spaces problem, how the pages were being crafted, and how they were linking to more and more of the autogenerated urls.

I also ran a smaller crawl using Screaming Frog. I crawled 50K urls just to see what our amphibious friend would surface. And again, you could see the autogenerated results rearing its ugly head. So from a crawling perspective, there was a lot of evidence of the page type spawning many low-quality urls. Next up was to check traffic levels to the site for that page type.

Before moving on, I exported a filtered list of the problematic pages from the crawl data. You can do this in both DeepCrawl and Screaming Frog. You’ll need those urls for a step later in the process. More about that soon.

Export filtered list of urls from DeepCrawl.

2) Google Analytics for trending and landing page data
My next move was to track trending over time to the problem urls to view how much traffic was being delivered there via Google organic search. I wanted to see how many of the pages were low-quality, how many were driving traffic versus the number indexed, was there serious volatility for the page type during major algorithm updates, and so on.

When checking trending overall, there was a clear drop from the February 7, 2017 update (which was a massive update). I wrote a post detailing that update, based on seeing a lot of volatility on sites across industries and countries. And when I isolated the problematic page type, the drop was even more distinct.

Problematic page type getting hammered during Google algorithm update.

To isolate landing pages from Google organic, you can access Acquisition, All Traffic, Source/Medium, and then click on Google/Organic. Then you can dimension by landing page. But note, Google provides sampled data in this report for larger-scale sites. Just keep in mind that you are not seeing all of the data.

Dimension by landing page in Google Analytics.

To view unsampled data, you can access Acquisition, All Traffic, Channels, Organic Search, and then dimension by landing page. But just keep in mind this report contains all organic search traffic and not just Google organic (although Google is a majority of organic search traffic for most sites).

Dimension organic search channel by landing page in Google Analytics.

When checking landing pages from Google organic to the problematic page type, I found only 7K urls that were driving traffic from Google organic. That’s out of hundreds of thousands indexed in just one area of the site. That’s a huge red flag by the way. Google doesn’t believe the content is strong enough to rank and drive traffic to, yet those pages are clogging up the index with many low quality urls. Remember, this problem could be impacting up to 10 million urls when you take the entire site into account.

Export Time:
Also, it was important for me to gather as many landing pages as possible, so I could trigger a subsequent crawl of those urls. Therefore, I exported all of the problematic landing pages from Google Analytics (from organic search) for further analysis. For larger-scale sites, I use Analytics Edge which enables me to tap into the API and export urls in bulk. It’s an amazing solution that I use heavily. I’ve also written several posts about it over the years.

Exporting landing pages from Google Analytics via Analytics Edge.

Optional Step: You can also run a medieval Panda report to see the drop after specific algorithm updates. That will enable you to see the landing pages from organic search seeing the largest drop after the update rolled out. You can often find glaring quality problems on those pages. You can learn more about running a Panda report in my tutorial. And no, it’s not just tied to Panda. It can be used for any traffic drop you are seeing.

Running a Panda report to compare traffic before and after an algorithm update.

3) GSC – The NEW Search Analytics Report (with 16 months of data)
The next step I took was to dig into the new search console reporting. All site owners now have access to the new GSC and it contains some extremely powerful reporting. It’s not complete yet, which is why you also have access to the old GSC at the same time.

First, you now have 16 months of data to analyze in the Performance reporting! That’s a new name for the Search Analytics report from the old GSC. Having 16 months of data is important and can help you view trending over a much longer period of time versus the old GSC (which was limited to just 90 days). You can also compare timeframes over a longer period of time.

If you have an infinite spaces problem, and you can isolate the page type via url structure, then the you can view trending of impressions and clicks over the past 16 months for the page type. That can also help you identify impact during major algorithm updates.

Filtering the Performance report in GSC by url.

Exporting via the Search Analytics API (Using Analytics Edge):
In addition, you can export ALL of your landing pages for that page type via the GSC API. Remember, we’ll need those urls in order to create a master list of urls. I’ll cover the surgical crawl in a later step. Note, the UI in the Performance Report and Search Analytics report only lets you export one thousand urls, which is limiting for larger-scale sites. But using a solution like Analytics Edge enables you to fully export your landing pages. Again, you can read my tutorial to learn more.

Just understand that the Search Analytics API only goes back 90 days for now. I believe Google is working on extending that to the 16-month timeframe soon. But at least you can get a feel for how many landing pages from Google organic fit under the problematic page type that’s creating infinite spaces.

Exporting landing pages from GSC via Analytics Edge.

4) Index Coverage – Excluded and Valid
As part of the new GSC, Google provides one of the most powerful new tools for site owners and SEOs. It’s called Index Coverage and it’s the old Index Status report on steroids. I’m part of the GSC beta group and I’ve been testing this for a while before the official launch. I can tell you, the reporting is packed with actionable data.

For example, you can drill into the pages that are valid and indexed, you can view errors by category, and you can view urls that have been excluded by Google by category. Yes, you are getting a rare view into how Google is treating urls and page types across your site. We have been asking for this type of reporting for a long time and the Index Coverage report delivers – big-time!

For our purposes today, we’ll focus on the Excluded and Valid categories. Since I knew there were many thin and low-quality pages due to the infinite spaces problem, I wanted to see how many were showing as being excluded by Google (essentially letting me know that Google decided to not index the urls for some reason).

It didn’t take long to see the problematic page type showing up in a number of reports. For example, the Soft 404s report had many of the urls in them. That’s because there were many urls being autogenerated without any content.

Excluded urls in the new Index Coverage report in GSC.

In addition, there were many showing up in the “submitted, but not indexed” category. Again, the site is providing the urls to Google (via sitemaps and via the internal navigation), yet Google has decided not to index many of them. It’s just more ammunition to take to your client, internal teams, etc. about the infinite spaces problem (if you need more ammunition).

Beyond the Excluded category, you can check the Valid category in the Index Coverage reporting and filter by the problematic page type to see how many of those pages are indexed.

Valid urls via the new Index Coverage report in GSC.

Export Time: You can export urls per category in the new GSC. That’s awesome, but there are limitations you should be aware of. First, you can only export up to one thousand urls per report (unlike the Search Analytics API, which lets you export the whole enchilada). I would have to check with Google again, but I believe they are working on adding API access for the Index Coverage report. I’ll update this post after finding out more.

Second, your filters don’t stick when exporting, and that’s problematic given what we’re trying to do here. Since we are filtering by page type, it would be awesome for the export to isolate those pages. But it doesn’t right now. All of the top one thousand urls are exported per category versus just the problematic page types. I submitted feedback to the GSC product team about this, so we’ll see if the export functionality changes. But for now, the Index Coverage report is for analyzing the page type, while the Search Analytics API can be used for full exports.

Export limitations in the new GSC.

5) Deduping landing pages, preparing for the surgical crawl
At this point, you should have analyzed trending for the specific page type and you have exported urls along the way. Now it’s time to create a master list of problematic urls so you can perform a surgical crawl. The surgical crawl will enable you to get a close-up view of the problem, along with any other issues plaguing that page type.

Excel is your friend for this task. Create a new spreadsheet and copy all of the worksheets you exported into that document. For example, you’ll use the Move or Copy functionality to copy the worksheet to our new document. Just right click on a worksheet tab and you’ll see “Move or Copy”.

Move and Copy in Excel.

Then create a new worksheet that contains all of the urls listed together. Since there will be overlap, you want to dedupe that list before crawling the urls. Click Data, and the Remove Duplicates. If you have a header row, make sure to check that box. Once you click OK, Excel will remove any duplicate urls. Now you have your final list for the surgical crawl.

Dedupe in Excel.

6) Perform a surgical crawl of JUST problematic urls
Depending on how many urls you have, you can choose to run the crawl via an enterprise crawler like DeepCrawl or via a desktop crawler like Screaming Frog. You could also choose to run a large-scale crawl of all the urls via DeepCrawl while crawling a subset of the urls via Screaming Frog (just to get a look at the urls via both tools).

Once the crawls complete, you can dig into the data to surface problems.

For example, you might (and probably will) find massive quality problems, thin content, empty pages, canonical issues, duplication, and more. And remember, this is just a subset of the total pages indexed for the problematic page type! Again, this is more ammo for your conversation with the various teams at your company or with your client.

Surfacing problems via a surgical crawl.

Next Steps – Preparing for battle and crafting your strategy
At this stage, you should have a boatload of intelligence about your infinite spaces problem. Every step you took will help you document the problem from several angles. And once you convey the situation to your client, dev team, c-level executives, etc., you need to tackle the problem head-on.

That includes deindexing as many of those urls as possible and then stopping the problem from surfacing again. That last part could include multiple steps, including changing functionality on the site and handling the urls via robots.txt. It depends on what you find based on your analysis.

For my situation, there was clearly a problem with the site autogenerating many urls. And only a small fraction of the urls are receiving traffic. Many of those pages are thin or low-quality. And the site can generate a near-endless list of these urls. I’m working with my client now on the final strategy, which will undoubtedly include deindexing many urls, refining functionality on the site that’s creating infinite spaces, and then disallowing urls from being crawled in the future.

But there are some important points to understand about this process. First, watch out for simply disallowing via robots.txt. I know many would move to do that immediately, but that’s not necessarily the best move right out of the gates. If you want to deindex the urls via noindexing or 404s/410s, which you should, then Googlebot wouldn’t be able to see the meta robots tag using noindex (or the 404s/410s) if you disallow via robots.txt.

I recommend keeping the directory open and letting Google deindex them over the long-term. Then you can disallow via robots.txt once they have been removed. That can take a while, so be patient. You can read my case study about removing a rogue subdomain to learn more about that.

Also, if the urls are in specific directories, then you could add those directories to GSC as a property. Then you could use the new Index Coverage and the old Index Status reports to monitor indexation. Many still don’t know that you can add directories as properties in GSC. I recommend doing that.

Monitoring indexation via Index Status in GSC.

XML sitemaps can help:
Also, you could submit xml sitemaps with all the problematic urls you collected and then check the new Index Coverage report for indexation, errors, etc. This is another benefit of exporting and deduping all of the problematic pages from DeepCrawl, Screaming Frog, Google Analytics, Search Analytics in GSC, and then Index Coverage in GSC. It’s totally fine to temporarily submit a sitemap that leads to 404s, 410s, or pages being noindexed. That can help Google with discovery and could lead to quicker deindexation.

Clarifying the Remove URLs Tool in GSC:
You can also choose to remove the urls via the Remove URLs Tool in GSC (especially if they are in a specific directory). But be aware that the Remove URLs tool does not work the way that many think it does. It’s a temporary removal (90 days) and only removes the urls from the search results (and not Google’s index).

Using the Remove URLs Tool in GSC.

You still need to actually remove the pages via 404, 410, or the meta robots tag using noindex for the pages to truly be removed. That’s extremely important to know, or you can end up in the same situation you were in when you started if you don’t handle the urls at the root. For an infinite spaces situation, you could temporarily remove the urls via GSC so they don’t show in the search results, while at the same time handle them on the site via 404s, 410s, or by using the meta robots tag using noindex.

Summary – Don’t let infinite spaces get in your way SEO-wise
Again, creating infinite spaces can be a dangerous thing from an SEO perspective, as Google can end up crawling a near-endless list of low quality, thin, or similar urls. It’s important to surface the problem and the urls being generated in order to properly tackle the situation.

Using a powerful SEO software stack, you can identify, analyze, and then address the problem efficiently. If you feel you have an infinite spaces problem (or might have one), I recommend following the process I provided in this post soon. Don’t let infinite spaces negatively impact your site SEO-wise. You can start today by “measuring infinity”. Good luck.

M-Dot Versus D-Top: How To Hack DeepCrawl’s Test Site Feature To Compare Your Mobile And Desktop Sites SEO-wise

January 23, 2018 By Glenn Gabe Leave a Comment

Comparing separate mobile urls to desktop using DeepCrawl's test site feature.

Last month I wrote a post covering a number of real-world mobile problems I surfaced on sites using separate mobile urls (like m-dot subdomains). With Google moving to a mobile-first index, it’s extremely important to make sure your mobile urls contain the equivalent content, structured data, canonical tags, hreflang tags, etc. as your desktop urls. Since Google will use the mobile urls and content for ranking purposes once the switch to m-first happens, you can end up with ranking issues and a drop in Google organic search traffic if your mobile urls are inadequate. And that’s never a good thing for sites looking to grow.

Now, if a site is responsive or using dynamic serving, it should be fine. But for sites using separate mobile urls (like an m-dot subdomain), there could be many gremlins lurking around. My post contains actual problems I have uncovered while helping companies prepare for Google’s mobile-first index. From botched switchboard tags to broken canonicals to double canonicals to rel alternate being delivered via the header response, companies could run into a number of problems that are tough to detect with the naked eye.

As part of my final recommendations in that post, I explained that it’s important to crawl your site as both Googlebot desktop and Googlebot for Smartphones to make sure the mobile crawl doesn’t surface many problems not present via the desktop crawl. It’s a great way to check your setup from both a desktop and mobile perspective. But I didn’t explain one tool that can really help those sites using separate mobile urls, and I’m going to cover that in this post. If you are using separate mobile urls, I think you’re going to dig it. Read on.

Comparing Desktop to Mobile Crawls (with the help of DeepCrawl)
You can definitely crawl your desktop urls and mobile urls and dig in manually. That’s a smart thing to do. But comparing larger-scale sites using this method can get tough. Wouldn’t it be great if a crawling tool could help by comparing the two crawls for you? Then you could surface problems faster and spend your time digging into those problems versus scanning thousands of urls.

Enter DeepCrawl’s test site feature, which you can hack to compare against your mobile subdomain.

Hacking DeepCrawl’s Test Site Feature To Compare Mobile Versus Desktop URLs
If you’ve read my previous posts about audits, migrations, algorithm updates and more, then you have come across many mentions of DeepCrawl. It’s one of my favorite SEO tools and has been for a long time. Actually, I loved it so much that I’m on the customer advisory board and have been a CAB member since it was launched a few years ago.

In DeepCrawl, there’s an option for crawling a test site. It’s been part of DeepCrawl for a long time and it’s a great way for SEOs to crawl and analyze a site while it’s in staging. Not only will it crawl a test site, but it will compare the test site to the production site. And if you’re following along with what I mentioned above about testing m-dots, then you might see where this is headed. :)

DeepCrawl's Test Site Feature

Although your m-dot is not a test site, you can hack DeepCrawl into thinking it’s a test site. Then it will crawl the mobile subdomain and compare to the previous crawl (which can be your desktop site). This technique was covered by Jon Myers in his presentation about preparing for Google’s mobile-first index. You should check out his presentation on SlideShare if you haven’t yet. And if you’re running a site that employs m-dots, then a lightbulb probably appeared above your head. Below, I’m going to explain how to set this up via DeepCrawl. Then I’ll cover some of the problems you can surface once the crawls have completed.

How to set up DeepCrawl to compare desktop and m-dot urls:
In order for DeepCrawl to compare separate mobile urls to desktop urls, you need an initial crawl of your desktop pages and then a second crawl of your “test site”, which will be your m-dot subdomain. Follow the steps below to set this up.

1. Set up a new crawl, enter your desktop domain and give the crawl a unique name.

Setting up a new project in DeepCrawl.

2. Check the box for “Website” crawl and do not check the box for “Crawl all subdomains”.

Selecting a crawl source in DeepCrawl.

3. Set the maximum number of urls to crawl. I recommend crawling as many as you can from the site (unless it’s a massive site). You want to make sure you are comparing apples to apples, so it would be optimal to get a significant sample from the site. For many sites, this won’t be a problem. If you have a large-scale site (greater than a few hundred thousand pages indexed), then you can craft strategies for crawling site sections, certain directories, etc.)

Setting crawl limits in DeepCrawl.

4. Next, click “Advanced Settings” which is marked with a gear icon. Again, make sure “Crawl subdomains” is not checked. We want to focus on the desktop site.

Uncheck crawl subdomains in the advanced settings in DeepCrawl.

5. Under “Link Restrictions”, uncheck “Crawl URLs in media mobile alternate tags” and “Crawl URLs in AMPHTML alternate tags. Again, we want the first crawl to be focused on our desktop urls.

Don't crawl mobile alternate tags or AMP tags in DeepCrawl.

6. Under “Spider Settings”, select Googlebot as the user-agent for the first crawl. In the second crawl, we’ll use Googlebot for Smartphones.

Selecting Googlebot as the user-agent in DeepCrawl.

7. Then click “Save and Start Crawl” to begin the crawl of your desktop urls.

Saving and starting your crawl in DeepCrawl.

8. {Optional} While the crawl is running, you can relax and sip a frozen drink while reading my tweets about SEO. :)

Dogs relaxing on a beach.

Crawl Two: Setting Up A Crawl Of Your Separate Mobile URLs (m-dot)
Once the first crawl completes, you’ll be ready to launch the second crawl (which will crawl your m-dot subdomain and enable DeepCrawl to compare the two crawls). Follow the instructions below to set this up:

1. From the main projects screen in DeepCrawl, click the edit icon (a small pencil icon). This will enable you to edit the settings so you can correctly crawl your mobile subdomain.

Editing the crawl settings in DeepCrawl.

2. Jump to the “Settings” screen, which is the fourth screen when editing your crawl settings. First, make sure “Compare to last crawl” is selected. The first crawl you launch won’t have a previous and that’s normal. But the second crawl you launch should be compared to the first (basically comparing m-dot to desktop). Note, you can always select a specific crawl to compare to in case you launched additional crawls of your m-dot and wanted to compare to a previous crawl of desktop or m-dot.

Comparing to last crawl in DeepCrawl.

3. Click “Advanced Settings” (fourth screen) again and jump down to “Spider Settings”. Select Googlebot for Smartphones as the user-agent for this crawl.

Selecting Googlebot for Smartphones as the user-agent in DeepCrawl.

4. Next, jump down to Test Settings and select “Test Site Domain and Custom DNS”. Under “Test site domain”, you should enter your m-dot subdomain on the first line. And since this isn’t a real test server (remember, we’re hacking this feature to compare the m-dot to desktop), you don’t need to enter anything in the Basic auth username or Basic auth password fields. Also, make sure “Use the test site with the crawl” is checked.

Hacking the test site feature in DeepCrawl by adding an m-dot.

5. That’s all you need to change in order for DeepCrawl to crawl your m-dot and then compare to the previous desktop crawl. Click “Save and Start Crawl” to begin the crawl of your m-dot.

Saving and starting a crawl in DeepCrawl.

6. {Optional}: Finish your frozen drink from earlier and read more tweets from me about SEO on Twitter. :)

More relexation at the beach while DeepCrawl runs.

Examples of problems you can surface when comparing m-dot to desktop:
Once the mobile crawl finishes, you’re ready to dig into the data and surface gaps between your desktop and mobile sites. The dashboard view will show you the number of top issues surfaced overall while also showing a trending line for each problem compared to the previous crawl.

DeepCrawl's dashboard surfaces top problems from the crawl.

When employing separate mobile urls, you need to watch out for major gaps and differences between your desktop site and mobile site. That includes content differences, canonical problems, internal linking gaps, crawl errors, hreflang problems, structured data issues, and more. You should review all of the major categories in DeepCrawl to make sure you have a solid view of how the crawls compare. Below, I’ll cover a few examples of what you can find.

Max Fetch Time (Good timing based on Google’s latest announcement, pun intended.)
In the screenshot below, you can see a major difference in the number of urls that took a long time to load between desktop and mobile. There were over 7K urls that had performance problems, including some urls taking up to 15 seconds to load.

Google just announced that mobile page speed will be a ranking factor as of July 2018, so this handy report can help you uncover mobile urls that might have performance problems.

Performance problems spike on the m-dot subdomain.

Thin content and empty pages
You might find more thin content or even empty pages in the mobile crawl when comparing to desktop. With Google’s mobile-first index, mobile pages will be used for rankings (once your site is switched to the m-first index). Therefore, you don’t want a situation where the desktop urls contains high-quality content, but some of the mobile urls are thin or even blank.

Empty pages on the mobile subdomain.

404s (Page Not Found)
You might find a spike in 404s on the mobile site, which can obviously cause big problems SEO-wise (if the pages should resolve with 200s and contain content matching the desktop alternatives). In this example, there’s definitely an increase in 404s on the m-dot, and the company will need to rectify the problems soon (as the mobile-first index approaches). And combine this report with the broken links report and you can hunt down which pages contain links to the 404s.

404s spike on the mobile subdomain.

Pages Without hreflang
For sites providing content in multiple languages, hreflang is a great solution. It enables you to tell Google which urls provide the content in which languages. Then Google can match up the correct page with the correct user based on language. When using hreflang, many site owners provide a “cluster” of alternate urls in the head of the document. You can also supply hreflang via sitemaps and via the header response, but many provide the tags directly in the code. If those tags are missing on your mobile urls, then hreflang will not work as expected. In the screenshot below, you can see a spike in pages without hreflang. There’s clearly a problem with the site publishing hreflang tags on the m-dot.

Hreflang problems on the mobile subdomain.

403s Spike on Desktop (Forbidden)
Well, that’s not good. You can clearly see a spike in 403s on the mobile site for some reason. You would clearly want to dig in and find out why that’s happening. That’s a sinister SEO problem that can definitely cause problems rankings-wise. I’ve seen some pages return 403s that load perfectly (so the urls are returning 403s, but they look like 200s).

403 errors on the mobile subdomain.

Missing Titles
When using separate mobile urls, it’s easy for some site owners to pay less attention to the m-dot. For example, not properly optimizing each mobile page. In the screenshot below, you can see a spike in missing title tags (where the title tags are completely missing from the page or empty). Again, when your site gets switched to the mobile-first index, the mobile urls and content will be used for ranking purposes. I would not have many urls without title tags when that’s the case.

Title tags missing on the mobile urls.

Broken Images
When comparing m-dot to desktop, you might also find problems with the images on your site. Again, many site owners neglect their m-dots as they focus on desktop. Below, you can see a spike in broken images on the mobile subdomain. Broken images could obviously impact image search, while also impact the user experience. It’s hard to have confidence in a site when the page looks broken. And it’s hard for images to rank in image search if they return 404s.

Broken images on the mobile subdomain.

I think you get the picture. As you can see, DeepCrawl can surface many issues when comparing m-dots to desktop sites (and many of those issues could be invisible to the naked eye). When you crawl at scale, you have a better shot at picking up those issues, which can help you fix important problems before your site is switched to Google’s mobile-first index.

Summary: Hacking is not always a bad thing (when DeepCrawl is involved)
The process I documented in this post covers a smart way to hack the “test site” feature in DeepCrawl to compare separate mobile urls to desktop. And based on Google’s mobile-first index, and Google’s announcement about mobile page speed becoming a ranking factor, it’s important to compare your mobile and desktop sites now (and not after the fact). Because if you compare them now, you can nip problems in the bud (which can help the site in the short-term and after Google’s changes go live). So hack away. :)

How To Quickly Remove A Rogue Subdomain From Google Using The Remove URLs Tool In GSC – And Then Make Sure It Stays Out (Case Study)

November 27, 2017 By Glenn Gabe 10 Comments

Deindex urls using the remove urls tool in GSC.

I’ve written about Murphy’s Law for SEO before, and it’s scary as heck. And that’s especially the case for large-scale websites with many moving parts. Murphy’s law is an old adage that says, “anything that can go wrong, will go wrong.” For example, no matter how much you plan and prep for large-scale SEO changes, there’s a good possibility that something will go wrong.

And when SEOs are implementing major changes across complex sites, whether that be CMS migrations, domain name changes, https migrations, etc., you can bet Murphy will be hanging around. You just need to be prepared for Murphy to pay a visit.

I recently helped a company move to https, and overall, it was a clean migration. I’ve helped this company with a number of SEO projects over the past several years and I was extremely familiar with their site, content, and technical SEO setup. The site was doing extremely well SEO-wise leading up to the migration, so we definitely wanted to make sure the move to https went as smoothly as possible.

We prepared heavily leading up to the migration, so we could all be confident when finally pulling the trigger. When a site moves to https there are a number of checks I perform prior to, during, and then after the migration takes place. And that includes several crawls to ensure everything is ok. In addition, my client is SEO savvy and they were checking the site from their end as well.

But remember our friend Murphy? Read on.

Site Commands Are Useless, Or Are They?
As we were tracking some of the normal volatility you can see during a migration, my client was checking indexation across some important urls. While performing some site commands, they noticed a rogue subdomain showing up with copies of the core website’s content. When I quickly checked that subdomain (via a site command) the subdomain revealed close to 7K pages indexed.

Site command revealing rogue subdomain indexed.

It seems when they pulled the trigger and moved to https, Murphy came to pay a visit. A subdomain that was never existent, suddenly appeared. And for some reason, the site was duplicating all pages from the core site on that subdomain.

Luckily, we were relatively early into the migration so we were able to start tackling the problem quickly. We clearly wanted to get those urls out of Google’s index as quickly as possible and then handle the subdomain properly. Below, I’ll document what we did to overcome a rogue subdomain showing up containing a lot of duplicate content.

Indexation, Rankings, and Traffic
One of the first things I told my client was to add that rogue subdomain to Google Search Console (GSC). That would enable us to see reporting directly from Google for that subdomain, including indexation, search analytics data, crawl errors, etc. It would also enable us to submit xml sitemaps and use the Removal Urls Tool if we thought that would be helpful. They quickly set up the GSC property and we waited for data to come in.

While we waited for data to arrive in GSC, I told my client to make sure all of the urls across the rogue subdomain 404. They did that pretty quickly. So the urls were only active for a relatively short period of time, but long enough for thousands to be indexed.

Once data arrived in GSC for the subdomain, we were able to see exactly what we were dealing with. There were 7,479 urls indexed according to Index Status reporting in GSC.

Surge in indexation via Index Status report in GSC.

In addition, some of the urls were actually ranking for queries and driving clicks. We saw 759 clicks and 28,724 impressions once the urls were indexed. Note, that was a small percentage of Google organic traffic compared to the core website, but clearly not a good thing that the rogue subdomain was driving impressions and clicks.

So we had a problem on our hands. Below, I’ll document the steps we took to remove the rogue subdomain from the Google search results while also working to have the pages removed from the index long-term. We also wanted to make sure that any subdomain urls that were ranking in the SERPs were correctly switched to the core site urls.

How To Remove Rogue URLs From Google’s Index:

1. 404 (or 410) all of the pages.
I mentioned this earlier, but I told my client to make sure all urls on the subdomain were 404ing. And yes, you can use 410s if you want. That could quicken up the process slightly for having the urls removed from the index. John Mueller has explained in the past that it could quicken up the process “a tiny bit”.
Here’s a video of John explaining that:

2. Verify the property in Google Search Console (GSC) – for several reasons.
We verified the subdomain in GSC so we could gain reporting directly from Google related to indexation, search traffic and impressions, xml sitemaps, crawl errors, and more. In addition, we would have access to the Remove Urls Tool, which can be very helpful with quickly removing urls from the Google search results (temporarily).

3. Create and submit an xml sitemap with all urls that are 404ing (via GSC).
We created an xml sitemap with all rogue urls and submitted that sitemap via GSC for the subdomain in question. That can help Google find the rogue urls that are now 404ing and hopefully get them out of the index quicker.
XML sitemap submitted with rogue urls.

4. Use the Removal URLs Tool in GSC for the new property.
In Google Search Console (GSC), there is a very powerful, yet dangerous, tool called the Remove URLs Tool. I find there’s a lot of confusion about how to use the tool, how it works, what will happen long-term, etc.

First, it’s important to know that it’s a temporary removal (90 days). Using the tool, you can temporarily remove urls from the Google SERPs (in case the urls contain sensitive information, confidential information, etc. that shouldn’t be displayed in the search results). This is probably the most common misconception. Some site owners remove a url or a directory using the tool and then think it’s gone forever. That’s not the case. You have to make sure you remove the urls via 404 or 410, or by noindexing the urls, in order for them to be truly removed from the index. You can also put the content behind a login (forcing a username/password to log in). If not, the urls can reappear 90 days later.

Second, you can remove specific urls, directories, or entire sites by using the tool in GSC. And the Remove URLs Tool is case-sensitive and precise (character by character). So make sure you enter the exact url when using the tool, or it might not work the way you want it to. Also, to remove an entire site or subdomain, just leave the form blank and click the button labeled “Continue”. All of the urls for the root GSC property you are dealing with will be removed.

The Remove URLs Tool in GSC.

Since we were dealing with a rogue subdomain, we wanted to nuke all of the urls residing there. So we simply left the field blank for the subdomain in GSC and pushed the giant nuclear button. Below you can see the subdomain removal request based on using the Remove URLs Tool.
Rogue subdomain removed from Google using the Remove URLs Tool in GSC.

Important Note: Make sure you are working with the right GSC property or you can incorrectly remove your core website from the SERPs. Remember I said the tool can be dangerous?

How Fast Will The URLs Be Removed?
That’s a trick question. There are two ways to think about the problem. First, how fast will the urls be removed from the SERPs? And then how fast before Google truly deindexes the urls (for the long-term). The Remove URLs Tool can work very quickly. Google’s John Mueller has said it can take less than a day for the urls to be removed from the SERPs, but I’ve seen it work even faster.

For this situation, it was just a few hours before a site command showed 0 results.
Site command revealing urls removed in just 3 hours.

But remember, it’s a temporary removal. You still need to handle the urls on the site correctly in order for them to be removed over the long-term. That means having the urls resolve with a 404 or 410, putting the content behind a login, or by using the meta robots tag using “noindex”. If you do one of those things, then the urls won’t reappear in Google’s index 90 days later.

Bonus: Double Check Logs To Ensure The Pages Aren’t Being Visited Anymore
And for those of you that dig analyzing log files, you can check your logs to make sure traffic drops off to the urls on the rogue subdomain. You can use Screaming Frog Log File Analyzer to import your logs and check activity (and hopefully see declining activity.)
Screaming Frog Log Analyzer for checking traffic.

Moving Forward: Monitoring The Problem Over The Long-Term
Once the above steps are completed, you shouldn’t just sit back with a frozen margarita in your hand gazing at Google Analytics. Instead, you should monitor the situation to make sure all is ok long-term. That includes reviewing the GSC reporting for the rogue subdomain, which can reveal several key things.

1. Make sure indexation drops.
Check the index status report and perform site commands to make sure the urls remain out of Google’s index. If you see them come back for some reason, make sure the urls are 404ing, they are properly noindexed, or behind a login. You should also make sure someone didn’t reinclude the urls in the Removal URLs Tool in GSC.
Index Status reveals urls being deindexed.

2. Make sure 404s show up in crawl errors.
By checking the crawl errors reporting in GSC, make sure the urls are showing up as 404s. For the subdomain we were working on, you can see the rise of 404s since my client implemented the correct changes. If you don’t see 404s increasing, you might not be correctly implementing 404s across the site or subdomain.
404s increasing as rogue urls are crawled by Google.

3. Make sure traffic drops off.
By checking the search analytics reporting in GSC, you can see impressions and clicks over time. Make sure they are dropping after removing the urls from Google. If you don’t see a drop in impressions and clicks for the rogue urls, then double check the Remove URLs tool, make sure the urls are indeed 404ing, being noindexed, etc.

Summary – Removing URLs… and Showing Murphy The Exit Door
If you find yourself in a situation where a rogue subdomain gets indexed with duplicate content, then use all of the tools available to you to rectify the situation. And that includes the Remove URLs Tool in GSC. Unfortunately, when implementing large-scale changes on complex sites, site owners should understand that things can, and will, go wrong. Murphy’s Law for SEO is real and can definitely throw a wrench into your operation. My advice is to be ready for Murphy to show up, and then move quickly to kick him out.

How To Use Facebook Pixel Reporting To Identify Who Is Secretly Using Fetch And Render In GSC On Your Site

August 3, 2017 By Glenn Gabe Leave a Comment

I can see you, and now you can see me. :)

A few months ago, Dan Sharp from Screaming Frog wrote a killer post about how to use fetch and render in Google Search Console (GSC) on almost any site. Fetch and render in GSC enables you see how Googlebot is rendering any page on your site (since it fetches all necessary resources to render the page at hand, including JavaScript, CSS, images, etc.) It enables you to detect differences in how Googlebot is viewing your page versus the average user.

It’s an awesome tool and can help identify blocked resources, trouble with rendering content, etc. If you want to learn more about what you can find using fetch and render, you can check out my Search Engine Land column titled Fetch and Horror.

Dan’s global fetch and render approach was to iframe urls from another unverified website on your own domain and then use fetch and render in GSC. It works, and pretty darn well. I’ve set this up and tested it myself, and it does indeed work.

For example, here’s a third party fetch of the Big Green Egg homepage (one of my favorite grills btw):

That’s cool, but let’s face it, rendering a site that you don’t own and control does give you a creepy feeling. You aren’t really authorized to use fetch and render in GSC on any site other than your own verified site(s). So rendering another person’s site is like virtually trying on their clothes from the comfort of your own home. Yes, that sounds creepy and it’s a good analogy. You are basically “fetching around”. You might feel like this:

And for those on the opposite end (being fetched), they can feel violated when someone is fetching and rendering their urls in order to see if there are any render problems. Again, it’s all a bit creepy.

But Are You Really Invisible?… No, Everything Is Trackable
As I was testing this out via my own setup, I started wondering how easily sites being rendered without consent could tell a third party was doing this. And if they could check that easily, it could lead to some uncomfortable situations. Which leads me to my next point…

Facebook Pixel Insights, With A Twist.
I was recently checking out Facebook’s pixel reporting and noticed an interesting finding. If you’re not familiar with the Facebook pixel, it enables you to cookie any visitor to your site and then retarget them back on Facebook. It’s a powerful approach that I’ve written about several times before (especially for boosting content to various custom audiences).

As I was reviewing the pixel reporting in Facebook Business Manager, I checked the domains tab. And clicking that tab just might blow your fetch-and-rendered mind. It will show you any domain where your pixel has fired. And I mean any domain.

In theory, most sites should only see their own domain listed… so if you see others listed, then something funny could be going on (like another site scraping your pages). And of course, this would show any site that is iframing your content. And that’s exactly what you need to do in order to fetch and render another site’s urls. Do you see where this is leading?

How To Access Pixel Reporting in Facebook Business Manager
In order to view your Facebook pixel reporting, first access Facebook Business Manager. Once you do, click the hamburger menu in the upper left corner. You’ll see several options in the menu, including Pixels. Click that link in the menu and you’ll be taken to a screen that shows all of your “pixel fires”.

Once you are viewing your pixel reporting, you’ll see four tabs, including Events, URLs, Domains, and Devices. The Domains tab is the one you should check out.

When checking the list of domains in my reporting, I noticed the test domain I was using to set up global fetch and render using Dan’s approach.

Here’s what I saw:

So my own domain is listed, which is correct, but there are several others there that shouldn’t be. And one is the domain I’m using for global fetch and render. Uh oh, I caught myself. :)

Beware, Your Digital Camouflage Isn’t Perfect:
OK, this means any site using the Facebook pixel could easily see that you have been fetch and rendering their urls. Cue creepy music. Remember, everything is trackable on the web (ok, almost anything). So, before you fire up your new global fetch and render setup, just know that websites have an easy way to identify who is fetching around. And by the way, who would think that Facebook would be the one helping SEOs uncover this? Ironic, don’t you think? :)

Big Recommendation: Set up FB’s tracking pixel (for more than this reason!)
When Dan’s global fetch and render approach was published, many were thrilled to try it out (myself included). But there were some concerns about who might be fetching your own site. Now with the Facebook pixel reporting, you can easily check the domains where your pixel is firing. And if any of those domains aren’t your own, then they just might be “fetching around”.

So definitely set up the Facebook pixel, and for more than just this reason. Boosting content to custom audiences is an amazing way to help get your content in front of a very targeted audience. And now that you can see who is rendering your own content, there’s yet another benefit to installing the Facebook pixel. So do it today. It should only take you a few minutes to set up.

In closing, fetch and render is powerful, but be careful if you’re “fetching around”. Facebook apparently has the back of site owners, and those site owners just might see you poking around. See what I did there? :)

How To Remove Rel Canonical From Noindexed Pages In WordPress Using The All In One SEO Plugin [Tutorial]

July 12, 2017 By Glenn Gabe 7 Comments

How to remove rel canonical from noindexed pages.

When performing SEO audits, it’s not unusual to surface pages being noindexed that also contain rel canonical. And that setup does’t make sense. Using the meta robots tag or x-robots-tag with noindex tells the engines to not index the page, while rel canonical tells the engines which is the preferred url for indexing. You can end up sending Google very confusing signals. Google’s John Mueller has explained this many times in the past and it could lead to some strange results.

Here are some links where John has mentioned this before:
https://www.seroundtable.com/noindex-canonical-google-18274.html
https://www.seroundtable.com/archives/020151.html

John Mueller explains that it’s best to not mix rel canonical and noindex (at 56:59 in the video):

And in an SEO world where certain things are out of your control, I’m a firm believer in taking control where you can. This is one of those situations. I always recommend that clients send the strongest signals possible about which pages should be indexed, and which ones shouldn’t be indexed. And providing both rel canonical and noindex definitely sends mixed signals.

SEO WordPress Plugins and Relinquishing Control
Many site owners using WordPress utilize SEO plugins to handle various tasks. For example, optimizing titles and metadata, managing indexation of different page categories, and more. But since the plugins are created by third parties, there are times it’s hard to customize your setup. And that can leave you with holes in your SEO armor.

One extremely popular SEO plugin, which I’ve used for a long time, is the All In One SEO Plugin, or AIOSEOP for short. It’s a plugin that has over three million active installs. There is also a pro version with additional functionality (and that’s what I’m running now). I’ve found the plugin to be extremely helpful for managing SEO tasks on WordPress sites.

I love the plugin, but for whatever reason, it always added rel canonical to noindexed pages. For example, if you noindex your categories or archive pages, rel canonical was still added. If you noindexed a certain post or page, rel canonical was added there too. This was definitely a thorn in my side, especially knowing that it’s best to not include rel canonical on noindexed pages.

It bugged me enough that I decided to find a solution.

Reaching Out To The AIOSEOP Team
I reached out to the All In One SEO Plugin team to see if there was a solution for handling this. I received an email back quickly with a nifty solution. It ends up you can use filter hooks to refine how the All In One SEO Plugin handles various situations. In a nutshell, by adding some code to your functions.php file, you can customize your setup. This is exactly the type of customization I was looking for.

So I fired up a text editor, opened my functions.php file and started testing some functions for handling rel canonical and noindex on certain page types. Below, I’ll cover how you can implement the same thing on your WordPress site.

How To Use Hooks To Remove Rel Canonical From Noindexed Pages
The code that handles this customization is pretty straight forward. You’ll add a filter via one line of code and then the function that removes rel canonical based on the condition that’s met. For my example, I wanted to remove rel canonical from any category or archive page, since those pages are currently being noindexed.

Here’s how to set this up:

Back up your WordPress blog. Any time you are going to mess with your WP code, you should absolutely back up everything. Make sure you can easily revert to the latest version. I automatically back up my blog daily and I also have my hosting provider backing up my entire site daily.
Open the functions.php file for your theme. It should be located in your theme’s root directory. If you have no idea what I’m referring to, then speak with your developer or hosting provider about finding the right file. If you want to learn more about functions.php, you can check out this article on wpbeginner.com.
Back up your current functions.php file before adding the new code. This is a great way to ensure you have a stable version of the file that you can revert back to, if needed. Simply download the file, save a backup, and then come back to the original file to customize the code.
Add the following code to your functions.php file. It checks if the current page is a category page or archive page and doesn’t write out rel canonical in the html if it is.

add_filter('aioseop_canonical_url','remove_canonical_url', 10, 1); function remove_canonical_url( $url ){ global $post; if( is_category() ){ return false; // Remove the canonical URL for categories. } elseif (is_archive() ){ return false; // Remove the canonical URL for archives. } return $url; }

Upload the functions.php file to your site (again, in your theme’s root directory). If you aren’t sure where this file should go, seek help first. Don’t just start uploading the file to different areas of your WordPress install.
Test the code by visiting your category pages or archive pages. If the code works, you should NOT see rel canonical being written out in the html. You should just see the meta robots tag using noindex.
You’re not done testing yet. Double check your homepage, your blog posts, and your pages to ensure rel canonical IS being written out properly. You want to make sure there’s not a conflict with other plugins or with your theme, which could result in some funky behavior.
If all looks good, then you’re all set! You can rest assured that your category and archive pages are not providing both rel canonical and the meta robots tag using noindex.

You Can Handle Noindexed Pages and Posts Too
Even though the code I provided above handles categories and archived pages, you can absolutely tailor the code to handle any page or post. You could provide additional conditions via the same function to remove rel canonical from specific pages or posts based on their ID.

For example, you could add the following code to the function we created earlier to check for a specific post. Just add another elseif that checks for a post ID:

elseif ( $post->ID === 2){ return false; // Remove the canonical URL for post #2. }

Summary – Rel canonical and noindex don’t mix.
If you’re using the All In One SEO Plugin, then this is a great solution for making sure rel canonical and noindex aren’t mixed together on the same page. Again, that can send mixed signals to Google. This technique can be used for major page types like categories or archives, as well as for specific pages or posts. Sure, you’ll need to edit some code, but it’s a flexible solution that shouldn’t take very long to set up.

Like I said earlier, don’t send mixed signals to Google. Take control and send clear signals instead.