Update: June 23, 2020
Disqus Indexing Bug Fixed By Google:
Google’s Martin Splitt confirmed there was a glitch that was inhibiting Google from indexing some Disqus comments. That glitch has been fixed and Google is now indexing Disqus comments. You can read more about that below in my post.
While helping companies that have been impacted by Google’s core updates, it’s obviously important to analyze content quality across each website. I’ve written heavily about the various quality issues that I’ve seen across sites negatively impacted by core updates and it’s hard to ignore the importance of what I call “quality indexing”. That’s ensuring your highest quality content is indexed, while making sure low-quality or thin content remains out of the index. Remember, Google has explained that it takes every page indexed into account when evaluating quality. You can watch this video or this one to hear Google’s John Mueller explain this.
So, it’s important to understand which pages Google is indexing, and which parts of a page are indexable. And for sites that allow user comments, I’m finding many site owners don’t realize how those blog or article comments contribute to the overall quality of each page, if they are indexable, and if those comments are actually being associated with the page at hand.
Also, it’s important to understand that comments help make up the content on the page, so you definitely want to know how user comments on your site are being handled by Google. That’s a good segue to the next section of this post.
Barry, John, And A Panda Walk Into A Bar…
Google’s John Mueller has said several times that blog comments help make up the content on the page. One of the first times we heard this is when Barry Schwartz asked John Mueller why Search Engine Roundtable had been negatively impacted by a medieval Panda update. John explained that low-quality comments can drag a page down (and the site overall as Google takes every page into account when evaluating quality).
So, high-quality comments can help your efforts, while low-quality comments can drag down the quality of the page (and site). Therefore, site owners should moderate blog and article comments heavily and ensure only high-quality comments that add value are published on a site.
In addition, Google’s Gary Illyes explained that (high-quality) comments and a thriving community flow into general quality. Those are extremely important points to understand from two different Googlers. Comments can help… and comments can hurt. That’s why it’s important to know how Google is handling the situation.
Comments In The Static Versus Rendered HTML
So, with user comments impacting content quality for a specific page (and contributing to the quality overall on a site), you definitely want a solid understanding of your own commenting setup, understand the ways to test what’s being rendered and indexed, and then implement changes to make sure you put your best foot forward from a quality perspective.
A Warning From The Field… Don’t Be Overconfident
Now, some of you might be extremely confident that you have amazing user comments and they provide a ton of value. That’s a great start, but are they really helping your site quality-wise? Unfortunately, you might be missing the second part of the equation – knowing if those blog or article comments are actually being indexed.
And as mentioned earlier, third-party commenting systems often load comments in an iframe. That doesn’t necessarily mean that content won’t be associated with your pages, but it does add another wrinkle to the situation. Google can choose to associate iframed content with your page, or not. Again, another reason to heavily test this out.
User Comments And The Difference Between Indexability and Indexing – A Process For Understanding Your Commenting System
Below I’ll provide several steps you can take to better understand if your blog or article comments are being indexed (along with various tools you can use). Remember, there are several factors involved here… so you might be surprised with what you find. Above all, you’ll at least understand if your comments are indexable and if they are actually being indexed (which are two different things).
And if your comments are being indexed, then it’s important to figure out if they should be… Remember, you should always moderate comments heavily and make sure those comments provide value to the user. If not, I would keep those out of Google’s index.
1. Are Your Comments Indexable?
The first step is to determine if Google can index your comments at all. For example, some sites might employ a process where the blog or article comments are in the static html. That’s the best way to make sure Google can see the comments and quickly index them. But that can require some custom development if you are using popular third-party commenting systems like Disqus.
The quickest way to check the static versus rendered html is to “view source” in Chrome, which will show you the static html. Then you can return to the page and right click and select “inspect”. That will show you the rendered html. You can search for your comments when checking both to see if the comments show up in the static html or just the rendered html. And you might even find the comments don’t show up in the rendered html if user action is required to view comments (like clicking a button to trigger the commenting system).
From a Google tool perspective, you can use the URL inspection tool in GSC to view the html for the crawled page and then test the live URL (which will show you the rendered html). So, you can test the live url and also see if the comments show up in the rendered html. As you can see below, they do. So again, Google should be able to index those comments. That doesn’t mean it will… but it can.
You can also use Google’s mobile-friendly test will show you the rendered html after fetching the url via Googlebot Smartphone.
And I find many people still don’t utilize Google’s Rich Results Test, which will show you either the desktop or mobile rendered html. This is great so you can check both Googlebot desktop and Googlebot smartphone user-agents (since the mobile-friendly test and URL inspection tool will only use the default crawler, which is Googlebot for Smartphones for any site moved to mobile-first indexing).
From a third-party tool perspective, Dave Smart developed a fetch and render tool that also renders iframed content (similar to how Google will).
Dave updated his tool after I reached out to see if it was rendering iframed content (since popular commenting systems are using iframes to display user comments). I’ve tested Dave’s tool heavily and it works extremely well. It’s just another tool in your arsenal when checking to see what’s in the rendered html (and what can be indexed by Google).
Note, I left out some third-party tools since they weren’t rendering iframed comments for some reason. Google can render those comments and sometimes indexes and associates the comments with the page at hand. So, to be safe, I would use one of the methods I listed above for testing. I’m going to reach out to those tools soon with my feedback and examples to hear their thoughts.
2. Indexable Doesn’t Mean Indexed – Know if your comments are being indexed.
OK, you now should have a better understanding if your blog or article comments are indexable. That’s the first step… but if you are moderating heavily and have high quality comments on your site, then you probably want those comments indexed and not just indexable. Google does not index all pages or every piece of content on a page. Google’s systems can choose to not index certain things based on a number of factors.
The best way to understand if your comments are being indexed is by using an old-school approach. Fire up Google and search specifically for those comments in quotes. You can mix that technique with a site command if you want to limit the search to your own site. But I would start by just searching for a line or two in quotes.
When doing this, you might be surprised with what you find. For example, I sometimes find that Google isn’t indexing comments for some sites when you thought it should be. Or, I might find that the comments are being indexed, but attributed to a profile page connected with the commenting system (like Disqus). Or, you might find that other pages from your site are being associated with the comments. Based on my research, that situation seems to be due to comment snippets being added in “related content” modules across other blog posts or articles (enticing users to click through to see those articles).
Regardless, this stage is super important. Don’t assume Google is simply indexing all indexable content. That’s not the case. Make sure you test your own site to see what Google is actually doing. If you have high-quality comments, make sure they are helping your pages (and site overall from a quality perspective). And if you have lower-quality comments, make sure you know if they are being indexed, and then how to stop that from happening if they are.
Here are some examples of what I’ve seen while researching the situation:
Native WordPress comments in the static html indexed:
More comments in the static html being indexed:
Disqus comments indexed and associated with the page at hand:
More Disqus comments indexed and associated with the page at hand:
Disqus comments indexed, but associated with a profile on disqus.com and not the actual blog post:
Disqus comments indexed, but associated with several other pages where the comments showed up:
Comments that require user action are indexed, but associated with another page on the site:
Facebook comments in the rendered html that are indexed and associated with the page at hand:
As you can see, the results are all over the board. This is why it’s so important to test your own situation heavily. Know what’s indexable, and then what’s actually being indexed by Google. You might be surprised with what you find…
**UPDATE: June 23, 2019**
Google Confirms That Disqus Indexing Glitch Has Been Fixed
Google’s Martin Splitt was asked on June 18th about Google possibly having issues with indexing Disqus comments (which I have seen during testing as well), and he said there was a glitch on Google’s end. That was super interesting to learn. He also said that we should watch this over the next few days since he was working with the indexing and rendering teams on fixing the issue.
Needless to say, I’ve been watching this closely. And today I started testing blog posts where the Disqus comments weren’t indexed. I used the URL inspection tool in GSC to check the crawled html vs. rendered html, and then checked to see if comments were indexed.
Well, I have really good news based on my testing this afternoon. It looks like Google is in fact indexing Disqus comments on pages that are being resubmitted for indexing! And I’m assuming that will be the case with normal crawling and indexing as well. Actually, Martin Splitt confirmed to me that the issue has been fixed!
This is incredibly important news for any site that uses Disqus for comments. If you have high quality comments, then those will now be indexed and associated with the page. And on the flip side, if you have low quality comments, then those will also be indexed… and be factored in as part of your content. So now is a really good time to review your comments from a quality standpoint.
Here are some screenshots of freshly indexed Disqus comments from my site:
You can also review the Web Story I created detailing the Google indexing problem with Disqus comments.
3. Publishing Comments In The Static HTML – Your Safest Bet
If your site has comments and you definitely want them indexed, then the safest approach is to make sure those blog or article comments are in the static html. The default comments in WordPress are published in the static html, so if you aren’t using a third-party commenting system, then you are probably in luck. But, if you are using a third-party commenting system like Disqus, then publishing comments in the static html will require some additional development work on your end. And not all third-party systems will support a local storage option.
4. Blocking User Comments From Being Indexed By Requiring User Action
One approach that some sites employ is requiring users to take action in order to see blog or article comments. That can also keep Google from seeing the comments since Google (typically) won’t click around a site.
For example, some sites will only show the comments after a user clicks a button to reveal them. That’s one way to make sure Google won’t see the comments in the static or rendered html. But again, test heavily and make sure your setup is working the way you want it to.
Final tips for understanding indexable versus indexed comments:
After reading this post, I’m sure you are eager to test out your own comments to see what Google is indexing versus not indexing. Remember, Google takes every page into account when evaluating quality and comments help make up the content on each page. Make sure you know what’s being indexed. Here are some closing tips:
- Moderate heavily. Blog and article comments help make up the content on the page. Don’t let low-quality comments on your site (especially spammy comments).
- Understand if comments are indexable… Remember, there’s a difference between indexable and indexed. Use the tools and techniques I provided in this post to see if the comments are located in the static or rendered html. That’s the first part of the equation.
- Next, find out if the comments are indexed. Use the techniques I listed earlier to see if Google is actually indexing the comments and associating that content with the page at hand. Again, you might be surprised with what you find.
- If you want your comments in the static html (which is the easiest way to ensure they get indexed), then you can use a local storage approach if you don’t want to rely on JS rendering. Check the documentation from the third-party commenting system you are using to see if this is an option. For example, Disqus provides a local storage option.
- If you want to block comments from being indexed, then you can choose to only show the commenting system via user action. Therefore, a user needs to act (like click a button) to reveal the comments. That can keep those comments hidden from Google (but make sure to test your own setup to make sure that’s the case).
- And of course, you can choose to not have comments at all. Many sites are going down this path and it is an option. If you can’t moderate heavily, then this might be a good approach. Then you don’t need to worry about what’s indexable or indexed.
Summary: Indexable Versus Indexed Comments
In this post, I covered the importance of understanding if your comments are indexable, and then if they are actually being indexed (which are two different things). Like many other topics in SEO, it’s critically important to test your own setup to see how Google is handling your user comments. Since comments help make up the content on the page, and each page is counted when evaluating quality for a site, it’s super important to understand what’s being indexed, and what’s not.
Now feel free to post a comment (see what I did there?) :)