In October of 2019, Google announced a plan to gradually block mixed content over time. Mixed content is a common issue across sites where secure pages attempt to load insecure content. For example, loading an image file over http from an https page. There are security risks associated with that, so Chrome wants to eliminate that risk in the future (by blocking mixed content from loading).

Since it’s a common issue, Chrome announced a staged plan in October so site owners would have plenty of time to research and fix mixed content issues on their sites. Even though the blog post was published in October of 2019, I’m still seeing a lot of confusion about the topic when working with clients. In addition, I’m finding many companies don’t even know that Chrome has begun blocking some types of insecure content already. 

In this post, I’ll first cover the timeline that Google mapped out for Chrome’s handling of mixed content. Then I’ll cover how the problem could break pages visually, I’ll cover the rendering impact, I’ll provide examples of what blocking looks like, and then I’ll end with some final tips.

Chrome’s Mixed Content Timeline:
In Google’s blog post about blocking mixed content over time, they provided a staged rollout. The plan was to gradually block mixed content versus simply flipping a switch and having pages break.

• The first change was in Chrome 79, where Chrome started providing a setting where users could unblock mixed content on specific sites. The setting applies to mixed scripts, iframes, and other types of content that Chrome blocks by default.
• In Chrome 80, which is the current version of Chrome as I write this article, Chrome will attempt to auto-upgrade audio and video resources to https (if possible). If the resources cannot be auto-upgraded, then Chrome will block the resources from loading. This is completed automatically by Chrome for sites trying to load audio or video resources over http from an https page.
• In addition, Chrome 80 will allow mixed images to load, but that will cause a “Not secure” label to be displayed in the browser. So, loading images over http will still allow the images to appear on the page (as of now), but the url will have a “Not secure” label in the browser window.
• Then in Chrome 81, which was supposed to be released in February 2020 but hasn’t rolled out, Chrome will attempt to auto-upgrade mixed images to https. If they fail to load over https, then Chrome will block those images completely (they won’t load on the page).
• Regarding the rollout of Chrome 81, the Chrome team announced that based on the global pandemic, Chrome 81 will not be released yet. Therefore, the final stage in the gradual rollout of blocking mixed content is not live yet. But, you can test it out via the Canary build of Chrome (which I’ll cover soon). As of today (Chrome 80), the stable version of Chrome is not auto-upgrading or blocking mixed images yet. It’s just showing a “Not secure” label for mixed images. But Chrome 80 is auto-upgrading audio and video resources to https and blocking them if they won’t load over https.

Insecure Images, A Common Mixed Content Issue
I’m sure some site owners and SEOs reading this post are confused about what mixed content is and how it looks. First, if you are running your pages over https, then all resources should be loaded over https. If you try to load resources over http, then that’s where mixed content warnings arise. Again, it’s a common issue that’s impacting many sites across the web.

As a simple example, here is a secure page (https) that is loading three images over http (insecure) in the stable version of Chrome (v80). Notice the “Not secure” label in Chrome versus the secure padlock.

When using Chrome Dev Tools and checking the security tab, you can view the resources that were loaded over http. You typically need to refresh the page to see those resources.

Since we are in stage three of the rollout (Chrome 80), the images will be allowed to load, but Chrome will display a “Not-secure” label in the browser window. To clarify, the images will still show up on the page and you will not run into any rendering issues (yet).  

It’s worth noting that audio and video will be auto-upgraded to https in Chrome 80. If they can’t be loaded over https, then the content will be blocked entirely. I’m focusing on images in this post, but that’s important to know if you are using audio and video on your pages.

Chrome Canary – The Ability To See Into Chrome’s Future
The Chrome Canary build is a version of Chrome that’s updated nightly with the latest functionality. It’s a great way to test your site and functionality for future versions of Chrome. There are many times I’m activating functionality in Canary that’s not yet in the stable version of Chrome to see how that will work when it finally does roll out. 

Since I know that Chrome 81 is nearing release, I was able to test how Chrome would handle mixed images. Remember, the stable version of Chrome simply adds a “Not secure” label in the browser window, but Chrome 81 will auto-upgrade those mixed images if possible. And if the images cannot load over https, then they will be blocked entirely. That means they won’t load on the page. And that also means it can impact what Google sees when it renders your pages.

As a quick example, I set up a page that loads an image via http and that image cannot be auto-upgraded to https. As you can see below, the image is blocked entirely from Chrome (and doesn’t show up on the page). The page actually shows up as secure in Canary (the “Not secure” label has been removed), but the page simply displays a missing image icon. That’s not good if it’s a critical piece of content for users.

Image blocked from loading:

The page is now secure, but the image is blocked:

The resource failed to load since it can’t be auto-upgraded:

Evergreen Googlebot, Broken Renders, and Ranking Impact
OK, so now we know that mixed images will not load in Chrome 81 if those images cannot be auto-upgraded to https. And we saw how that looks in the browser with broken image icons showing up on the page. That’s not good for users at all, but what about Google?

Google’s John Mueller has explained that since Google uses an evergreen Googlebot now for rendering, it will be using the latest version of Chrome. That means it will be using Chrome 81 once that rolls out. Again, that hasn’t rolled out yet due to the global pandemic, but it will at some point soon.

So if Googlebot will be using Chrome 81 to render your content, and we already know that Chrome 81 will block mixed images, then that’s exactly how Google will see it. So key images will be missing from your renders if those images cannot be loaded over https.

In addition, from a ranking perspective, John explained that the blocking of images shouldn’t impact rankings based on Google’s https ranking signal. That’s good to know, but he also explained that if the pages cannot render completely in a browser, then Google probably won’t be able to render those parts either. And he clearly recommended fixing that issue.

Here is a link and screenshot to a reddit thread where John explained this:

Wrapping Up and Final Tips:
Before I end this post, I wanted to provide a number of final tips and suggestions based on researching this topic. Remember, site owners still have time before Chrome 81 rolls out to address mixed content issues. Based on what I provided in this post, it would be smart to rectify those issues now before Chrome 81 rolls out. Here are some final tips:

• Crawl your site and surface all mixed content issues. My three favorite crawling tools are DeepCrawl, Screaming Frog, and Sitebulb. All three provide specific reporting for mixed content. Note, I’m on the customer advisory board for DeepCrawl. It has been one of my favorite enterprise crawling tools since 2013.
• Understand that Chrome 81 has NOT rolled out yet and Google has paused updates due to the global pandemic. Chrome 80 is the stable version of Chrome now and it does display a “Not secure” label for pages with mixed content problems. Also, Chrome 81 will roll out relatively soon and will completely block mixed images that don’t load over https (when auto-upgraded).
• Chrome 80 currently attempts to auto-upgrade audio and video to https. And it those resources fail to load over https, Chrome will block them from loading.
• Mixed content problems can impact your pages visibly for users, and for Google rendering your pages. Remember, Google now uses an Evergreen Googlebot, which utilizes the latest version of Chrome to render your pages. That means Googlebot will be using Chrome 81 to render pages once it rolls out. As Google’s John Mueller explained, you definitely want to fix mixed content issues that impact rendering.
• John Mueller also explained that this shouldn’t impact rankings based on Google’s https ranking signal. That’s good news, but again, it can impact what users see on your pages and what Google is able to render.
• You can use Chrome Canary to test your pages now to see how mixed content will impact your site once Chrome 81 rolls out. It’s a great way to test Chrome’s latest and greatest functionality that hasn’t rolled out to the stable version of Chrome yet.

Address Mixed Content Issues and Avoid Broken Pages and Partial Renders
In this post, I covered Chrome’s plan to gradually block mixed content. The final stage of the rollout will be completed when Chrome 81 is released as the stable version of Chrome. But as I covered in my post, the release of Chrome 81 has been delayed due to the global pandemic. Therefore, you still have time to fix mixed image problems before Chrome blocks them from loading (if they can’t be auto-upgraded to https). And remember, audio and video resources are already being blocked if they can’t be auto-upgraded.

I recommend thoroughly testing your site to ensure you surface, and then fix, mixed content problems. Then you can rest assured that both users and Googlebot can load and render all of your resources, including images, video, and audio.

GG

For many site owners and SEOs, SafeSearch can be a confusing topic. For example, understanding how SafeSearch works, what filtering looks like in action, and how it can impact traffic from Google organic search. That’s especially the case for sites that walk a fine line between providing non-explicit and explicit content (at least according to Google’s algorithms).  

I’ve worked on several cases recently where SafeSearch came into play. Site content was being filtered for users with SafeSearch active, which was impacting the visibility of content in the SERPs and subsequent traffic to the site.

But these weren’t porn sites being filtered… far from it actually. Instead, the sites are publishers that write about a range of topics within their niche. And sometimes those articles provide content and images that trip the SafeSearch filter. When that happens, those urls and images are filtered out of the search results yielding lower traffic levels from Google.

And to clarify, filtering applies to images, videos, and websites, so it’s not just about filtering in the standard organic results. It impacts your media as well.

This is why it’s a smart idea for publishers to become as knowledgeable as possible about Google’s SafeSearch feature, how it works, how to test if you’re being actively filtered, and how to test new content to future-proof your site.

That’s because if you trip the filter, traffic could drop to various degrees depending on how the filter is applied. For example, is SafeSearch just impacting certain areas of the site or the entire site? It depends on your setup, your content, and how far your content crosses the line.

Also, while helping companies deal with SafeSearch filtering, I’ve received many questions about how to test content to see how close it is to tripping the SafeSearch filter. And if you do trip the filter, how do you know you are being filtered? And what about if you are incorrectly being filtered? Can you reach out to Google to let them know that’s happening?

These are all important questions and I’ll be covering a number of these topics in this post.

Images and Text, Not Just Explicit Photos
A lot of people don’t know this, but Matt Cutts was tasked with creating the first version of SafeSearch. He explained more about this during a recent interview with Barry Schwartz (at 12:08 in the video). In that interview, Matt explained that Google initially used text to understand what should be filtered. Since the first iteration of SafeSearch was in about 2000, Google didn’t have machine learning algorithms to help identify pornographic images. Therefore, it was mostly text-based.

SafeSearch is undoubtedly much more sophisticated now and Google has explained it does use machine learning to detect explicit content that should be filtered. More about that soon, but I wanted to make sure you understood that both text and images are taken into account. Also, there’s a SafeSearch team at Google that is focused on the feature. That’s also important to understand in case you run into trouble, like if your site is incorrectly being filtered.

An older Google help document I surfaced, which is now removed, explains that SafeSearch looks at “keywords, links, and images” to determine what should be filtered. So again, it’s not just images.

Tripping The SafeSearch Filter – What Can Happen?
When a site trips the SafeSearch filter, Google will filter some (or all) of your pages for users that have SafeSearch active (again, across images, video, and websites). It’s an option in your search preferences, so some people have to manually turn it on. For others, it might be turned on at a higher level, like a school administrator, IT department, or even a parent managing a child’s computer.

If this happens, then your pages won’t appear like they normally would for certain queries leading to your site. They will be filtered out. Poof, gone. There’s no communication from Google about this, like a message in GSC (at least that I know of). Instead, your pages will just be filtered in the SERPs.

As you can guess, some sites can experience lower traffic levels depending on their target demographic. For example, if your audience includes people that have SafeSearch turned on, then the filter can cause problems traffic-wise. If most users that visit the site don’t have SafeSearch active, then you might not even know you are being filtered. It’s tricky that way.

But let’s face it, “tricky” is a four-letter word in SEO. As a site owner or SEO, you definitely want to know if your content is being filtered by SafeSearch. You don’t want any surprises down the line, like ghostly traffic drops that are hard to figure out. That can be very frustrating. I’ll explain how to check for SafeSearch filtering soon, but first a note about site structure and adult metadata.

How Site Structure Can Help Google Focus Its SafeSearch Filtering
Google has explained that you can help its algorithms understand which sections contain adult content by using a specific url structure for organizing content (see the section about optimizing for SafeSearch at the bottom of that page). For example, placing all adult content in a specific directory.

If you do that, then Google’s algorithms can determine that the entire site shouldn’t be filtered, but just the specific directory. That can save you down the line, so I highly recommend trying to organize your content that way if you need to publish adult content for some reason (or content that Google believes is explicit).

Now, you might be wondering what exactly constitutes “adult content”. That’s a tricky one… since it’s determined algorithmically. I’ll cover ways to understand how your content is being viewed by SafeSearch later in this post.

Adult Meta Tags
Google also explains that the best way to let them know that adult content is on your page is to use adult metadata. They say it’s the “strongest signal” you can provide about this. There are two meta tags you can use in your code to accomplish this (you just need to include one, but you have a choice).

How To Check For SafeSearch Filtering – Tripping The Filter
Below, I’ll cover several ways to check to see if your content is actively being filtered by SafeSearch. You’ll be checking after-the-fact, meaning you are already being filtered. After I cover these methods, I’ll explain how to conduct some preventative work to make sure you stay out of the danger zone. i.e. Testing content you want to publish to see how Google’s SafeSearch sees the content.

Before we begin, I find it’s easier to check for SafeSearch filtering with two browsers open. One will have SafeSearch on and the other off. And if you have dual monitors, it’s great to have one browser up on each display.

1. Perform a site command with SafeSearch active
First, fire up the browser with SafeSearch on and perform some site commands (including by directory). In extreme cases, you will see 0 results returned when your entire site is flagged and filtered.

For sites not completely filtered, I’ve found site queries to be a little sketchy. Sometimes they show a site being partially filtered and sometimes they don’t… So use this as a quick way to check for filtering, but don’t 100% rely on it. Again, for sites being completely filtered by SafeSearch, this can be helpful. For partial filtering, this process can be unreliable.

2. Identify at-risk queries and perform actual searches
The best way to check for SafeSearch filtering is to start searching using the two browser-setup I mentioned earlier. First, go to GSC and export a bunch of queries leading to your site. Find ones that would be considered risky from a SafeSearch perspective (either based on the query itself or the content you have at the destination landing pages). And then start searching.

Enter a query in the browser with SafeSearch off, check your rankings, and then perform the same search in the browser with SafeSearch on. You might be surprised with what you find. You might see major gaps in the SERPs where your pages used to rank. If that’s the case, yep, you’ve tripped the filter.

I just did this for a client and found many queries that yielded SafeSearch filtering for them. And the filtering was focused on one specific directory (which made complete sense considering how they have the site structured).

Remember what I said earlier about the importance of url structure for SafeSearch? Well, they luckily house all risky content in one directory. That’s the good news. The bad news is that there’s a lot of non-explicit content there as well… So, certain pages are being filtered by SafeSearch, but at least it’s not the entire site being filtered.

Repeat for more queries and identify which queries and landing pages are being filtered. By analyzing more of this data, you can start to get a feel for what crosses the line content-wise. That’s a good segue to the next section of this post… how to catch problematic content before it gets filtered.

How far is too far?… Enter Google’s Vision API
Up to this point, I’ve covered how to check if you’re actively being filtered, but what about performing some preventative work BEFORE you get filtered? This is a natural question for any company that has gotten caught in the filter. They normally want to future-proof their site and they don’t want Google categorizing the site as adult (or a site that houses some adult content).

Well, Google has you partially covered, pun intended. :) Enter Google’s Vision API, which is pretty amazing. It uses machine learning to automatically identify and categorize images. You can read more about the Vision API on the site, but I’ll stick to how we can use it based on this post.

One of the features of the Vision API is that it can be used to detect explicit content via SafeSearch. It can identify adult content, violence, and more. It also uses OCR to detect text in the image, so it’s not purely about the images in the photo.

Also, if you have development resources, then you can tap into the API at scale to check your images using SafeSearch. If you don’t have development resources, and you’re not checking a lot of images on a regular basis, then you can test specific images right on the site. There’s an option to try the API for single images.

After dragging an image to the tool, or selecting one from your computer, Google’s Vision API will analyze that photo and return a boatload of information, including sections for faces, objects, labels, entities, webpages that contain the image, text, properties, and… SafeSearch.

Yes, there’s a dedicated section for SafeSearch that provides the likeliness that the image is adult, spoof, medical, violence, or racy. It’s fascinating.

When testing various images using the API, you can start to get a feel for what crosses the line. Here are some examples of what the API returns for SafeSearch. The first returned the likelihood of being “racy”, but not really “adult”, while the second crossed the line and returned “very likely” for both “adult” and “racy”.

Note, although you’ll receive some great information from the API, it’s hard to tell exactly what will trip the SafeSearch filter. That said, this sure gives you a stronger view of how SafeSearch works. If you see “Very Likely” for both Adult and Racy, then that’s clearly an image you want to rethink (if you don’t want to trip SafeSearch). Also, you don’t want Google to begin to classify your site, or parts of your site, as having adult content.

False Positives, Notifying Google That Your Site Is Incorrectly Being Flagged
Machine learning algorithms aren’t perfect and there are times sites are incorrectly flagged as having adult content. If you experience this situation, then you should let Google know about it. In the past there was a form you could fill out to let Google know your site was incorrectly being flagged by SafeSearch.

Unfortunately, Google removed that form and now drives site owners to the webmaster forums to make their case. I’m not a big fan of that approach since site owners often don’t want to announce to the world that their site is being flagged by SafeSearch (especially if they don’t focus on adult content…)

So, the good news is that you can post in the forums and hopefully get your information in front of a Googler on the SafeSearch team. The bad news is that you have to tell the world about it, since the post is going to be public.

Feature Request: SafeSearch Report in Google Search Console Wanted!
After helping some companies with this situation, I think it would be great to have a notification in GSC or some type of report that lets site owners know if their content is being flagged by SafeSearch. And maybe have a “request review” button there in case the site, or sections of the site, are incorrectly being flagged. Here is what that can look like (hint, hint GSC team). :)

Closing Tips: SafeSearch, Tripping The Algo, and Future-proofing your site
As I explained earlier, it’s important to become as familiar as possible with SafeSearch if you provide content that could be classified as explicit (like adult content). Using the methods and tools listed in this post can help you determine if you are currently being filtered, and to understand if you might be filtered in the future. Below, I’ll cover some final tips and recommendations:

• Understand your current situation fully. If you think SafeSearch might be at play with your own site, perform thorough testing to see if you are being filtered. Then form a plan of attack if you are. Use the methods I listed earlier to test queries with SafeSearch off, and then on. Again, you might be surprised with what you find.
• If you provide adult content (or any type of content that might be deemed explicit), then start conversations with your marketing and dev teams about site structure. Remember, Google has explained that placing all adult content in a specific directory can help its algorithms understand where explicit content resides on your site (and then SafeSearch can be as granular as possible).
• Use adult meta tags to provide self-marked pages that contain explicit content. Google explained this is the strongest signal you can send to them that your pages contain adult content.
• Identification-wise, SafeSearch takes both images and text into account. It’s not just the image content.
• If you are being filtered, then that filtering can happen for webpages, video, and images. It’s not just about filtering in the standard organic search results.
• Educate your editorial staff about SafeSearch and show them how to test images via Google’s Vision API. This can help protect your site down the line by having employees conduct preventative checks BEFORE that content is published.
• And if you believe your site is being incorrectly flagged by SafeSearch, you unfortunately can’t submit a form to tell Google like in the past. You must go got the webmaster forums to provide the details. That can hopefully get in front of a Googler on the SafeSearch team, but there are no guarantees.

Summary – Try To Be Safe With SafeSearch
I hope this post helped you gain a better understanding of Google’s SafeSearch feature, how to test your own site, and then how to conduct preventative checks. Don’t let some ghostly drop in traffic create a confusing situation for your company. Know when you’re being filtered and then form a plan for handling that situation. Good luck and safe searching :)

GG

Update: June 23, 2020
Disqus Indexing Bug Fixed By Google:
Google’s Martin Splitt confirmed there was a glitch that was inhibiting Google from indexing some Disqus comments. That glitch has been fixed and Google is now indexing Disqus comments. You can read more about that below in my post.

———————————–

While helping companies that have been impacted by Google’s core updates, it’s obviously important to analyze content quality across each website. I’ve written heavily about the various quality issues that I’ve seen across sites negatively impacted by core updates and it’s hard to ignore the importance of what I call “quality indexing”. That’s ensuring your highest quality content is indexed, while making sure low-quality or thin content remains out of the index. Remember, Google has explained that it takes every page indexed into account when evaluating quality. You can watch this video or this one to hear Google’s John Mueller explain this.

So, it’s important to understand which pages Google is indexing, and which parts of a page are indexable. And for sites that allow user comments, I’m finding many site owners don’t realize how those blog or article comments contribute to the overall quality of each page, if they are indexable, and if those comments are actually being associated with the page at hand.

Also, it’s important to understand that comments help make up the content on the page, so you definitely want to know how user comments on your site are being handled by Google. That’s a good segue to the next section of this post.

Barry, John, And A Panda Walk Into A Bar…
Google’s John Mueller has said several times that blog comments help make up the content on the page. One of the first times we heard this is when Barry Schwartz asked John Mueller why Search Engine Roundtable had been negatively impacted by a medieval Panda update. John explained that low-quality comments can drag a page down (and the site overall as Google takes every page into account when evaluating quality).

So, high-quality comments can help your efforts, while low-quality comments can drag down the quality of the page (and site). Therefore, site owners should moderate blog and article comments heavily and ensure only high-quality comments that add value are published on a site.

In addition, Google’s Gary Illyes explained that (high-quality) comments and a thriving community flow into general quality. Those are extremely important points to understand from two different Googlers. Comments can help… and comments can hurt. That’s why it’s important to know how Google is handling the situation.

Comments In The Static Versus Rendered HTML
Some sites I’ve helped have many comments per page (dozens and sometimes even hundreds per article). Some of those comments are present in the static html (and easily indexable), while others may require JavaScript rendering (and will only show up in the rendered html). And then there are some third-party commenting systems where the comments are contained in an iframe (which provides its own unique situation). All of this has led to confusion about what’s actually being indexed from a comments standpoint.

Regarding rendering, in the past you could add comments via JavaScript and feel relatively safe that Google would not see those comments. Well, things have definitely changed over the past several years. Google has done a lot of work with indexing JavaScript content and JS-based comments can often be indexed (if that content is present when the page loads and doesn’t require user action). Google’s Martin Splitt has heavily covered its two-wave approach to indexing, which you can see below (static + rendered html).

So, with user comments impacting content quality for a specific page (and contributing to the quality overall on a site), you definitely want a solid understanding of your own commenting setup, understand the ways to test what’s being rendered and indexed, and then implement changes to make sure you put your best foot forward from a quality perspective.

A Warning From The Field… Don’t Be Overconfident
Now, some of you might be extremely confident that you have amazing user comments and they provide a ton of value. That’s a great start, but are they really helping your site quality-wise? Unfortunately, you might be missing the second part of the equation – knowing if those blog or article comments are actually being indexed.

When helping clients dig into this situation, I’ve noticed some very interesting things. For example, some comments are not being indexed at all, some require JavaScript rendering, some only load after user action, sometimes the comments are being indexed on third-party pages (other domains), and sometimes Google is able to index the comments, but just doesn’t choose to do so. I’ll show some examples of this later in the blog post.

And as mentioned earlier, third-party commenting systems often load comments in an iframe. That doesn’t necessarily mean that content won’t be associated with your pages, but it does add another wrinkle to the situation. Google can choose to associate iframed content with your page, or not. Again, another reason to heavily test this out.

User Comments And The Difference Between Indexability and Indexing – A Process For Understanding Your Commenting System
Below I’ll provide several steps you can take to better understand if your blog or article comments are being indexed (along with various tools you can use). Remember, there are several factors involved here… so you might be surprised with what you find. Above all, you’ll at least understand if your comments are indexable and if they are actually being indexed (which are two different things).

And if your comments are being indexed, then it’s important to figure out if they should be… Remember, you should always moderate comments heavily and make sure those comments provide value to the user. If not, I would keep those out of Google’s index.

1. Are Your Comments Indexable?
The first step is to determine if Google can index your comments at all. For example, some sites might employ a process where the blog or article comments are in the static html. That’s the best way to make sure Google can see the comments and quickly index them. But that can require some custom development if you are using popular third-party commenting systems like Disqus.

So, fire up a blog post or article with comments and get ready to check the static and rendered html. This will enable you to understand what’s indexable from the start versus requiring JavaScript. There are several tools you can use to check the static versus rendered html. Some will do this in bulk while others just check a single page.

The quickest way to check the static versus rendered html is to “view source” in Chrome, which will show you the static html. Then you can return to the page and right click and select “inspect”. That will show you the rendered html. You can search for your comments when checking both to see if the comments show up in the static html or just the rendered html. And you might even find the comments don’t show up in the rendered html if user action is required to view comments (like clicking a button to trigger the commenting system).

From a Google tool perspective, you can use the URL inspection tool in GSC to view the html for the crawled page and then test the live URL (which will show you the rendered html). So, you can test the live url and also see if the comments show up in the rendered html. As you can see below, they do. So again, Google should be able to index those comments. That doesn’t mean it will… but it can.

You can also use Google’s mobile-friendly test will show you the rendered html after fetching the url via Googlebot Smartphone.

And I find many people still don’t utilize Google’s Rich Results Test, which will show you either the desktop or mobile rendered html. This is great so you can check both Googlebot desktop and Googlebot smartphone user-agents (since the mobile-friendly test and URL inspection tool will only use the default crawler, which is Googlebot for Smartphones for any site moved to mobile-first indexing).

From a third-party tool perspective, Dave Smart developed a fetch and render tool that also renders iframed content (similar to how Google will).

Dave updated his tool after I reached out to see if it was rendering iframed content (since popular commenting systems are using iframes to display user comments). I’ve tested Dave’s tool heavily and it works extremely well. It’s just another tool in your arsenal when checking to see what’s in the rendered html (and what can be indexed by Google).

Note, I left out some third-party tools since they weren’t rendering iframed comments for some reason. Google can render those comments and sometimes indexes and associates the comments with the page at hand. So, to be safe, I would use one of the methods I listed above for testing. I’m going to reach out to those tools soon with my feedback and examples to hear their thoughts.

2. Indexable Doesn’t Mean Indexed – Know if your comments are being indexed.
OK, you now should have a better understanding if your blog or article comments are indexable. That’s the first step… but if you are moderating heavily and have high quality comments on your site, then you probably want those comments indexed and not just indexable. Google does not index all pages or every piece of content on a page. Google’s systems can choose to not index certain things based on a number of factors.

The best way to understand if your comments are being indexed is by using an old-school approach. Fire up Google and search specifically for those comments in quotes. You can mix that technique with a site command if you want to limit the search to your own site. But I would start by just searching for a line or two in quotes.

When doing this, you might be surprised with what you find. For example, I sometimes find that Google isn’t indexing comments for some sites when you thought it should be. Or, I might find that the comments are being indexed, but attributed to a profile page connected with the commenting system (like Disqus). Or, you might find that other pages from your site are being associated with the comments. Based on my research, that situation seems to be due to comment snippets being added in “related content” modules across other blog posts or articles (enticing users to click through to see those articles).

Regardless, this stage is super important. Don’t assume Google is simply indexing all indexable content. That’s not the case. Make sure you test your own site to see what Google is actually doing. If you have high-quality comments, make sure they are helping your pages (and site overall from a quality perspective). And if you have lower-quality comments, make sure you know if they are being indexed, and then how to stop that from happening if they are.

Here are some examples of what I’ve seen while researching the situation:

Native WordPress comments in the static html indexed:

More comments in the static html being indexed:

Disqus comments indexed and associated with the page at hand:

More Disqus comments indexed and associated with the page at hand:

Disqus comments indexed, but associated with a profile on disqus.com and not the actual blog post:

Disqus comments indexed, but associated with several other pages where the comments showed up:

Comments that require user action are indexed, but associated with another page on the site:

Facebook comments in the rendered html that are indexed and associated with the page at hand:

As you can see, the results are all over the board. This is why it’s so important to test your own situation heavily. Know what’s indexable, and then what’s actually being indexed by Google. You might be surprised with what you find…

**UPDATE: June 23, 2019**
Google Confirms That Disqus Indexing Glitch Has Been Fixed
Google’s Martin Splitt was asked on June 18th about Google possibly having issues with indexing Disqus comments (which I have seen during testing as well), and he said there was a glitch on Google’s end. That was super interesting to learn. He also said that we should watch this over the next few days since he was working with the indexing and rendering teams on fixing the issue.

https://twitter.com/g33konaut/status/1273654384489046016

Needless to say, I’ve been watching this closely. And today I started testing blog posts where the Disqus comments weren’t indexed. I used the URL inspection tool in GSC to check the crawled html vs. rendered html, and then checked to see if comments were indexed.

Well, I have really good news based on my testing this afternoon. It looks like Google is in fact indexing Disqus comments on pages that are being resubmitted for indexing! And I’m assuming that will be the case with normal crawling and indexing as well. Actually, Martin Splitt confirmed to me that the issue has been fixed!

This is incredibly important news for any site that uses Disqus for comments. If you have high quality comments, then those will now be indexed and associated with the page. And on the flip side, if you have low quality comments, then those will also be indexed… and be factored in as part of your content. So now is a really good time to review your comments from a quality standpoint.

Here are some screenshots of freshly indexed Disqus comments from my site:

You can also review the Web Story I created detailing the Google indexing problem with Disqus comments.

3. Publishing Comments In The Static HTML – Your Safest Bet
If your site has comments and you definitely want them indexed, then the safest approach is to make sure those blog or article comments are in the static html. The default comments in WordPress are published in the static html, so if you aren’t using a third-party commenting system, then you are probably in luck. But, if you are using a third-party commenting system like Disqus, then publishing comments in the static html will require some additional development work on your end. And not all third-party systems will support a local storage option.

As a quick example, Disqus provides a local storage option which enables you to database your Disqus comments and then provide them in the static html. So, each page will contain your comments without requiring JavaScript rendering (and without requiring Google to make decisions about iframed content).

4. Blocking User Comments From Being Indexed By Requiring User Action
One approach that some sites employ is requiring users to take action in order to see blog or article comments. That can also keep Google from seeing the comments since Google (typically) won’t click around a site.

For example, some sites will only show the comments after a user clicks a button to reveal them. That’s one way to make sure Google won’t see the comments in the static or rendered html. But again, test heavily and make sure your setup is working the way you want it to.

Final tips for understanding indexable versus indexed comments:
After reading this post, I’m sure you are eager to test out your own comments to see what Google is indexing versus not indexing. Remember, Google takes every page into account when evaluating quality and comments help make up the content on each page. Make sure you know what’s being indexed. Here are some closing tips:

• Moderate heavily. Blog and article comments help make up the content on the page. Don’t let low-quality comments on your site (especially spammy comments).
• Understand if comments are indexable… Remember, there’s a difference between indexable and indexed. Use the tools and techniques I provided in this post to see if the comments are located in the static or rendered html. That’s the first part of the equation.
• Next, find out if the comments are indexed. Use the techniques I listed earlier to see if Google is actually indexing the comments and associating that content with the page at hand. Again, you might be surprised with what you find.
• If you want your comments in the static html (which is the easiest way to ensure they get indexed), then you can use a local storage approach if you don’t want to rely on JS rendering. Check the documentation from the third-party commenting system you are using to see if this is an option. For example, Disqus provides a local storage option.
• If you want to block comments from being indexed, then you can choose to only show the commenting system via user action. Therefore, a user needs to act (like click a button) to reveal the comments. That can keep those comments hidden from Google (but make sure to test your own setup to make sure that’s the case).
• And of course, you can choose to not have comments at all. Many sites are going down this path and it is an option. If you can’t moderate heavily, then this might be a good approach. Then you don’t need to worry about what’s indexable or indexed.

Summary: Indexable Versus Indexed Comments
In this post, I covered the importance of understanding if your comments are indexable, and then if they are actually being indexed (which are two different things). Like many other topics in SEO, it’s critically important to test your own setup to see how Google is handling your user comments. Since comments help make up the content on the page, and each page is counted when evaluating quality for a site, it’s super important to understand what’s being indexed, and what’s not.

Now feel free to post a comment (see what I did there?) :)

GG

Google has done a lot of work over the past two years with Google Images. And all signs point to Google driving forward with even more new features and functionality. Although sometimes an afterthought for many companies when it comes to Search, Google Images can drive meaningful traffic. In addition, having your images properly indexed can lead to stronger listings in Google’s Top Stories (if you’re a news publisher).

Therefore, if you have strong visual assets, and you know people are searching for those images, then it’s important to give your site the best shot possible to rank for relevant queries.

By the way, I attended the Google Webmaster Conference in Mountain View on Monday, November 4, and the presentations from the various Googlers were awesome. One “lightning round” covered Image Search and the various projects Google has been working on recently.

An important point covered in that presentation related to “Google moving from just showing images to providing much more context about the great content behind those images”.

That has manifested itself with a new design that includes titles and descriptions directly in image search, swipe up with AMP to visit a site, and more. In the past, only images would be displayed, so Google is definitely trying to help users find the content behind the images, which can translate to higher-quality image traffic to your site. So again, it’s important to rank for the images you want to rank for.

That’s a good segue to the case study I’m presenting in this post. :)

Case Study: When One “Top Stories” Listing Is Not Like The Others – The Image Advantage
I received an email one morning from a company with an interesting question. The news publisher was experiencing a weird problem with Google Images, including images in Top Stories. The publisher performed well in Google News, including Top Stories, but images would not show up in the Top Stories listing a good percentage of the time. Sometimes they did, but often they didn’t. And they were ultra-confused about why that was happening.

They explained that there were images in the articles and you could clearly visit those images and see them perfectly when visiting the CDN urls (like many companies, they were using a CDN to store and deliver their images). But when their articles ended up ranking in Top Stories (an extremely visible search feature that can drive a ton of traffic), images often did not accompany the article.

And to make matters worse, the other sites listed in the same Top Stories module did have images, which put this company at a disadvantage from a click-through rate standpoint. Here’s a mockup of what it looked like:

So, I dug in to hopefully provide some answers. And like many other things in SEO, what’s invisible to the naked eye was causing serious (but logical) problems. Read on.

So what the heck was wrong? The land mine of extra CDN hops…
As I mentioned earlier, the company was using a CDN to house and deliver images. That’s totally fine for handling images and many sites employ CDNs for that very purpose. Even John Mueller has fielded that question a number of times and explained it’s totally fine to use CDNs for that.

But the devil is in the details. Sure, a CDN is fine, but you need to follow the path that Googlebot must follow in order to see if it’s truly fine. So I crawled part of the site and began checking the situation manually (via Chrome and several plugins). It wasn’t long before I could see the problem.

Tracking the path from the site to CDN revealed an extra hop along the way (via a redirect). And that extra hop was extremely problematic. It was through another CDN subdomain… which had me immediately checking the robots.txt file on that subdomain. And low and behold, the urls were blocked by robots.txt.  

So the image urls sent Googlebot to a CDN url, which then sent it to another CDN url (which was blocked by robots.txt), and then to the final CDN url where the image was housed. Since Googlebot couldn’t crawl the extra hop, it was never making its way to the image file. And that’s why images weren’t being indexed, ranking in Google Images, or showing in Top Stories. Not good, to say the least.

To the naked eye (for non-SEOs), you could enter the image url and see that the image was displayed fine. But, the path to that image was problematic. Again, it didn’t take long to see the 301 to 301 to 200 redirect chain. And then checking the second 301 more closely, I cross referenced the robots.txt file for that domain, and boom, it was blocked.

The good news is that this was a relatively straightforward fix (although many of the image urls across a large and complex site needed to be refined). So, the company’s dev team dug in and worked on a better solution. Again, it’s a large and complex site, so no change is easy… But they clearly wanted to move quickly to get this done.  

Side note: It ends up not all images were set up this way across the site. That’s one of the reasons the company was confused about why images weren’t showing up in Top Stories. Some of the images were being indexed fine in Google Images. That’s when the extra hop wasn’t present. This made it harder for the company to debug. For example, they saw some images being indexed fine, and showing up in Top Stories, but many weren’t.

The issue was sort of sinister that way… almost cloaking the bad results with some good.  That said, many images in the latest news articles were having problems in image search (and Top Stories). So that led them to start asking the right questions, which led to the root problem that I covered above.

A fix was implemented and… image traffic surged!
The dev team worked on a better solution and removed that extra hop. Now Googlebot is able to crawl and index the images efficiently and associate those images with the pages at hand. I was eager to see how Google would handle the change. And I was pretty excited with the results.

It’s like Google was just waiting for the fix… Google image search traffic surged and has continued to increase over time. You can clearly see trending for image search surge after the fix was put into place in April. Those metrics continue to move up and to the right. Here are some screenshots of the surge over time:

Since the change, an extra 334K clicks have been driven from image search alone. Unfortunately, it’s hard to gauge the changes in traffic from Top Stories, since there’s no way to specifically report on that in GSC across desktop and mobile, but articles are showing up with images now versus just text (which should be helping with click-through rate).  

For Top Stories, this change levels the playing field for my client. The other articles showing up from other news organizations often had images associated with them, which can greatly help with click-through rate. Now the company’s Top Stories listings also have visuals. It’s great see.

Driving Google Image Performance – Closing Tips
Google Images can definitely drive meaningful traffic, and images can be an important part of Top Stories (which can also drive a lot of traffic for news publishers). As I mentioned earlier, Google has made a serious effort to enhance image search over the past two years or so, and I’m sure more is coming on that front. So, it’s important to make sure you are providing a clear path for crawling and indexing your image content (so that content can end up ranking in Search).

Here are some final bullets containing important image search tips:

• In order to rank in Google Images, Google needs an image and landing page combination. Images alone will not suffice. Google’s John Mueller has covered this several times.
• Don’t block Googlebot from crawling your images, whether they are hosted on your own site, on a subdomain, or housed on a CDN. And make sure you follow the full path to those images, which could include extra hops that might be blocked by robots.txt.
• Crawling your site on a regular basis using tools like DeepCrawl, Screaming Frog, and Sitebulb can help you surface redirect chains and various files blocked by robots.txt. And they do this in bulk. When you combine your toolset with your brainset, good things can happen. :)
• Check the actual search results and use tools to view screenshots of SERP history to ensure your listings and images look ok. For example, perform actual queries that are yielding Top Stories and view your listings. You can also use tools like SEMrush to view historical screenshots of the SERPs, which can also help you understand how your listings looked over time. All of this can help you understand what’s going on, but can also provide data to stakeholders when a problem exists. As they say, a good image is worth a thousand words. Well, a good SEO screenshot is worth a thousand votes when you need changes implemented.
• Unfortunately, sitemap reporting in GSC does not contain the number of images indexed anymore. John Mueller once said that’s probably a good thing to have back, so maybe we’ll see that at some point in the future. For now, you really don’t have a solid idea of how many images are being indexed or the problems that could be causing indexing issues. Maybe we’ll see a new sub-report in GSC’s coverage reporting for this in the future… who knows? (Yes, that’s a hint for GSC product managers and engineers!) :)

Summary – Let your images be seen (and crawled and indexed properly)
In closing, a redirect chain combined with a robots.txt directive was causing major problems with image search (and Top Stories) for the news publisher that contacted me. With Google’s focus on improving image search, you don’t want to be left out in the cold if you provide barriers to Googlebot when crawling and indexing your image content.

Sometimes the answer lies in the path to those images… so make sure you double check redirects, redirect chains, and if those urls are accessible to Googlebot.  There just might be some obstacles in Google’s way. And that’s typically not a good thing.

GG