The Internet Marketing Driver

Google just rolled out another broad core algorithm update on June 3 (which was preannounced by Google’s Danny Sullivan.) And once again, the core ranking update was big. It wasn’t long before you could see significant impact from the update across sites, categories, and countries. Some sites surged, while others dropped off a cliff. And that’s par for the course with Google’s core updates.

For example, here are three examples of drops from the June 2019 Google Core Update:

But I’m not here to specifically cover the June update. Instead, I’m here to cover an extremely important topic related to all broad core ranking updates – conducting user studies. It’s something I have mentioned in a number of my posts about major algorithm updates, and Googlers have mentioned it too by the way. More on that soon.

My post today will cover the power of user studies as they relate to core ranking updates, and provide feedback from an actual user study I just conducted for a site impacted by several major updates. By the end of the post, I think you’ll understand the value of a user study, and especially how it ties to Google’s core updates by gaining feedback from real people in your target audience.  

Google: Take A Step Back And Get Real Feedback From Real People:
After core updates roll out, Google’s John Mueller is typically pummeled with questions about how to recover, which factors should be addressed to turn things around, etc. And as I’ve documented many times in my posts about core updates, there’s never one smoking gun for sites negatively impacted. Instead, there’s typically a battery of smoking guns. John has explained this point many times over the years and it’s incredibly important to understand.

But beyond just taking a step back and surfacing all potential quality problems, John has explained another important point. He has explained that site owners should gain objective feedback from real users. And I’m not referring to your spouse, children, coworkers, top customers, etc. I’m talking about feedback from objective third parties. i.e. People that don’t know your site, business, or you before visiting the site.

When you conduct a study like that, you can learn amazing things. Sure, some of the feedback will not make you happy and will be hard to take… but that’s the point. Figure out what real people think of your site, the user experience, the ad situation, the content, the writers, etc. and then form a plan of attack for improving the site. It’s tough love for SEO.

Here is one video of John explaining that site owners should gain feedback from objective third-parties (at 13:46 in the video). Note, it’s one of several where John explains this:

Conducting User Studies Through The Lens Of Google’s Core Updates:
When you decide to conduct a user study in order to truly understand how real people feel about a site, it’s important to cover your bases. But it can be a daunting task to sit back and try to craft questions and tasks for people that will capture how they feel about a number of core site aspects. As I explained above, you want to learn how people really feel about your content-quality, the writers, the user experience, the advertising situation, trust-levels with the site, and more. So, crafting the right questions is important.

But where do you even begin??

Well, what if Google itself actually crafted some questions for you? Wouldn’t that make the first user study a lot easier? Well, they have created a list of questions… 23 of them to be exact. And they did that in 2011 when medieval Panda roamed the web.

The list of questions crafted by Amit Singhal in the blog post titled More guidance on building high-quality sites provides a great foundation for your first user study related to Google’s core algorithm updates.

For example, the questions include:

• Would you trust the information presented in this article?
• Is this article written by an expert or enthusiast who knows the topic well, or is it more shallow in nature?
• Would you be comfortable giving your credit card information to this site?
• Does the article provide original content or information, original reporting, original research, or original analysis?
• Does the page provide substantial value when compared to other pages in the search results?
• How much quality control is done on content?
• And more…

As you can see, these are incredibly important questions to review. The questions can absolutely help you better understand how real users are experiencing your site, how they feel about your site, and ultimately, the questions can help craft a remediation plan covering what you need to change or improve on your own site.

I have used these questions (or variations of them) to run both quick and dirty user studies, and formal studies. The feedback you can receive is absolutely gold. Not just gold, but SEO gold in the age of broad core ranking updates.

Let’s face it, this is exactly the type of information that Google is trying to evaluate algorithmically. So, although it’s not easy to run user studies, and it can be time-consuming and tedious, it’s one of the most important things you can do as a site owner.

Beyond The 23 Panda Questions, More Ideas From The Quality Rater Guidelines (QRG)
The Panda questions provide a great foundation, but you can absolutely run more user testing using Google’s Quality Rater Guidelines (QRG) as your foundation. And there are a boatload of topics, ideas, and questions sitting in the 166-page guide that Google uses with its own quality raters.

I won’t spend too much time on the QRG in this post, but some topics you might test include:

• Page quality.
• Needs-met ratings.
• Usability.
• Monetization and Advertising.
• Authors, writers, review boards.
• Site reputation.
• E-A-T.
• User intent.
• And more…

Now, you can just trust me (and John) and think that user testing is important, or you might want more information. For example, like seeing examples of what you can really learn from a user study. Well, I’ve got you covered. I just conducted a user study for a site that was heavily impacted by the March core update (and that has seen major volatility during several core updates over the years). The feedback we received from the user study was awesome and I’m going to share some of it with you (without revealing the site). I think you’ll get the power of user studies pretty quickly.

User Testing Results: What You Can Learn From Real People: Health/Medical Case Study
Again, the site has seen big swings (up and down) during various core updates and I’ve been helping them identify all potential quality problems across the site (including content-quality, technical SEO, user experience, advertising situation, site reputation, UX barriers, and more).

After fully auditing the site, I used the Panda questions mentioned earlier as the foundation for the user study and tailored some of those questions for the niche and site. Below, I’ll provide some of things we learned that I thought were extremely important for my client to understand. Remember, this is real feedback from real people.

Test-wise, I not only used multiple choice questions, but I also used open-ended questions to learn more about how each user felt about certain situations. In addition, I used a platform that provides session recordings of each user going through the study. For this study I used usertesting.com and I’ll explain more about testing platforms later in this post.

I can tell you that watching and listening to people experience a site is absolutely fascinating. There is so much you can learn from hearing the reaction of users, picking up things they say, and watching how they navigate a site or page.

So, the combination of quantitative feedback, qualitative feedback, and viewing recorded sessions provides the ultimate recipe for surfacing potential problems on a site. And that feedback can directly help site owners craft a remediation plan that goes beyond fixing minor issues. Instead, you can start to address deeper issues and problems. And that’s exactly what Google’s core updates are about… Google is evaluating a site overall and not just looking at one or two factors. Remember, there’s never one smoking gun.

First, some quick background information about the user study:
By the time I was setting up the test, I had already fully analyzed the site and provided many areas for improvement. But, we wanted to gain feedback from real users in the site’s target audience about a number of important topics. Also, I wanted to use the 23 Panda questions as a foundation for the test.

Audience Selection:
Since usertesting.com has a panel of over one million people, I was able to select specific demographic information that enabled us to make sure the test participants were part of my client’s target audience. For example, I was able to select gender, age, household income, if they were parents (and how old their children were), job status, web expertise, and more. I’ll cover more about this later.

So, what were some things I wanted to learn from the participants? Here are a few of the things I was interested in:

• Did users trust the information provided in several articles I asked them to read?
• Did they think the articles were written by experts, or just people heavily interested in a topic?
• Was the content original? Or did they think it could easily be found elsewhere on the web?
• Did they recognize the brand? How about the founders and writers?
• How did they feel about recency, original publication dates, if the articles were updated, and how that was treated on the page?
• I asked them to review and provide feedback about the background and experience of the site owners, authors, and the medical review board.
• I wanted to know if the participants thought there was an aggressive, disruptive, or deceptive advertising situation (since this was a problem when I first started analyzing the site).
• And more… there were 39 different questions and tasks I had the participants go through.

Below, I’ll cover some pieces of feedback that we thought were extremely helpful. By the way, some of the responses (and video clips) were eye-opening. I’ll provide the details below.

Examples of feedback from the user study (in no specific order):

• Balance – Several participants mentioned the importance of balance in the article. For example, thoroughly covering the benefits AND risks of certain topics. Again, this is something that can be very important in articles, especially YMYL articles.
• Triggers – I learned that certain words were triggers for some people, which I could only hear in the video clips. I would have never known that from multiple choice questions. For example, when certain words were read aloud, some participants would react in a way that clearly showed how they felt about that topic. They even said, “whenever I read {enter word here}, that immediately throws up a red flag for me”. Wow, amazing feedback for the site owners.
• Sources and Credibility – Along the same lines, the sources and citations were extremely important for some of the participants. Some explained that if they see Wikipedia as a source, they immediately become skeptical. One even said it discredits the article. For example, one user said, “wait, so it’s reviewed by a doctor, but it cites Wikipedia… not sure I trust this article at all.”
• Trust & Reactions – When asked about if a certain participant trusted one of the articles, she laughed out loud. Again, hearing people in the video is incredibly powerful. And laughing is typically not a good thing for a YMYL site. :)
• Publish Dates – There were several important pieces of feedback regarding publish dates, updated dates, etc. First, some assumed that if there was an updated date on the article, then that meant the entire article had been fully reviewed again. That can be deceptive, since the articles just had specific pieces updated.
• More About Publish Dates – Some participants absolutely wanted to see the original publish date and the updated date. They did not just want the updated date, since that makes them search for clues about when the article was originally published. Some participants explained the process they go through to find the original publish date, which included checking the sources being cited (and the dates associated with those sources). And then they use a savvy approach of checking the comments for dates.
• Social Proof – I heard one participant explain how if she sees a lot of comments, then that means it must be a popular website. Very interesting… Comments are tough for many sites due to the onslaught of spam, the time involved in moderating comments, etc., but they do seem important for some people.
• Author Expertise – Several participants wanted to know the background of the writers as they were reading each article. Since the articles they were reading covered health topics, they immediately went into “skeptical mode”. This was important to see and underscores the importance of having experts write the content.
• Citing Sources – Several participants explained that just a link to a source wasn’t enough for some articles. They wanted to see stats and facts backing up some claims (in the article itself). For example, maybe providing some of the data directly in the article versus just linking out to another article.
• “Just A Blog…” – I heard several remarks comparing blogs to medical websites. For the health niche, this was very interesting feedback. There was a negative stigma with blogs for some users, especially for health/medical topics.
• Advertising Situation – Advertising-wise, there were also some interesting pieces of feedback. Remember, there was an aggressive advertising situation when I first started helping the client, so I was extremely interested in hearing what the participants thought of the current ad situation (which has improved, but the site owners haven’t moved as far as I would like them to). I heard one user literally counting the number of ads as she scrolled down the page. 1, 2, 3, wait more, 4, 5. But in a strange twist, she then said the ad situation was fine… She knew there were a number of ads, but didn’t find them distracting. It’s extremely important to make sure the advertising situation is ok, since Google has explained that aggressive ads can impact a site algorithmically over time.
• Affiliate Marketing – Regarding affiliate links, I did hear, “Are they just trying to sell me something?? Ok, they probably are…” This is something I have brought up to my client during the audit and it’s a tough conversation to have. But remember, Google has explained that there’s a fine balance when delving into affiliate links or affiliate marketing in general. There must be a lot of value added versus monetization. If the scale tips in the wrong direction, bad things can happen Google-wise. So this piece of feedback was extremely important to see/hear directly from users.
• Author Expertise – When asked about the expertise of the author of an article, the user started scrolling to find the author information and then said, “Wait, it’s a blog… no, I don’t trust the author at all.” I heard this type of comment several times during the user study. More about building a brand and credibility soon.
• Content-Quality – When asked about original content across the articles, almost all of the users in the study said there was some original content, but some of it could easily be found in other places across the web. Not one person said the content was original. This underscores the importance of tackling subject matter where you can provide original content, ideas, perspectives, etc. If you write about what many others are writing about, the content can be viewed as quasi-original. That’s not good enough for a tough niche.  
• Content Value – When asked about substantial value from the content compared to other articles on the topic, every one of the users said it was average compared to the others. You clearly don’t want to strive for “average”. You want 10X content. This was great for my client to see. They have strong articles overall, but users saw them as average compared to the competition.
• Side note: SERP UX – When watching users go to Google and look for a competing article, it was fascinating to see several scroll right by the featured snippet and select something a little farther down the page (in the standard organic results). Sure, this isn’t a large sample size, but just an interesting side note.
• Site Design – When researching other articles on a topic, a user commented that all the sites look the same. And those sites ranged from some of the top health sites on the web to academic sites to health blogs. Site design, branding, etc. comes into play here and it’s something that I don’t think many focus on enough.
• Brand Recognition – Regarding brand, every one of the users in the study said they never heard of the site, brand, etc. This is clearly a signal that the site owners need to work on branding. For example, getting the brand out there more via PR, reaching eyeballs beyond their core audience, etc.
• Recency – For health topics, I heard a user explain they definitely want to see more recent articles on a topic. The article they were reading was a few years old and that didn’t seem sufficient for her. Recency seemed important (but it must actually be recent and not just an “updated on XX” tag slapped on the page).
• Affiliate marketing – More comments about “they are advertising {enter product here}” while reading an article. So yes, users pick up on affiliate links. Again, the value from the article must outweigh the monetization piece.
• Citing sources – There were positive comments about certain sources that were cited, like consumer reports, a scientific study, etc. For health articles, I saw users in the video checking the sources at the bottom of the page, which could help build credibility.
• Medical review board – Overall, the users liked that articles were reviewed by a medical review board. I heard this several times while reviewing the recorded sessions of participants reading the articles.
• Expertise and Credibility – When asked about the expertise and background of the site owners, authors, and medical review board, there were plenty of interesting comments. For example, having a medical review board with various types of doctors, nutritionists, etc. seemed to impress the participants. But I did hear feedback about wanting to see those credentials as quickly as possible on the page. In other words, don’t waste someone’s time. Don’t be too cute. Just provide the most helpful information that builds credibility as quickly as possible.
• Awards and Accolades – For various awards won, users want a link to see more information about that (or they wanted to see more on the page itself). It’s clearly not good enough in this day and age to simply say you won something. Let’s face it… anyone can say that. They want proof.
• Trust – When asked if they would be comfortable giving their credit card information to the site, most responded, “I’m not sure I would go that far…” or “No, definitely not”. So, there were clearly some breakdowns with trust and credibility. I saw this throughout various responses in the study. My client has some work to do on that front.
• UX barriers – I noticed errors pop up twice while reviewing the video clips of users going through the site. If these are legit errors, then that’s extremely helpful and important to see. I passed the screenshots along to my client so their dev team could dig in. It’s just a secondary benefit of user testing (with video recordings of each session).
• And there were many more findings…

As you can see, between reading their responses, hearing their reactions, and then watching each video session, we gained a ton of amazing feedback from the user study. Some of the feedback was immediately actionable, while other pieces of feedback will take time to address. But overall, this was an incredible process for my client to go through.

User Testing Platforms – Features & User Panel
If you just read the sample of findings above and are excited to conduct your own user study, you might be wondering where to start. Well, there are several important things to consider when preparing to launch a user study. The first is about the platform you will use.

Usertesting.com is probably the most well-known platform for conducting user studies and it’s the one I used for this test. I was extremely impressed with the platform. The functionality is killer and their panel of over one million people is outstanding.

In addition, participants sign a non-disclosure agreement (NDA), which can help reduce the chance of your test getting shared publicly. Some sites wouldn’t care about this, but others would care. For example, I know a number of my clients would NOT want the world knowing they are running a user study focused on trust, quality, advertising situation, etc.

Audience-wise, I was able to select a range of criteria for building our target audience for the user study (as covered earlier). This enabled me to have participants that were closely tied to my client’s target audience. It’s not perfect, but can really help focus your audience.

Functionality-wise, you can easily create multiple choice questions, open-ended questions, etc. You can also use Balanced Flow to send users through two different test flows. This can enable you to test different paths through a site or different customer experiences.

Here are some screenshots from the test creation process:

Pricing-wise, usertesting.com isn’t cheap… but could be well worth the money for companies that want to perform a number of user tests (across a range of actions). Remember, the sky’s the limit with what you can test. For example, site design, usability, features, content-quality, site trust, and more. I was ultra-impressed with usertesting.com.

Beyond usertesting.com, I also looked into UsabilityHub (Google is a client of theirs btw) and userlytics. I have not used these other platforms, but they could be worth looking into since they also have large panels of users and what seems to be strong features.

Closing Tips and Recommendations:
Before ending this post, I wanted to provide some closing tips and recommendations when setting up your first test. I am by no means an expert on user testing, but I have learned some important lessons while crafting tests:

• First, user testing is not easy. It can be time-consuming and tedious (especially when analyzing the results). Build in enough time to craft your questions and flow, and then enough time for fully analyzing the results. You might be surprised how much time it takes to get it right.
• For Google’s core updates, you can definitely use the 23 Panda questions as a foundation for your test. You also might take a subset of those questions and then tailor them for a specific niche and site. After that, you can use the Quality Rater Guidelines as a foundation for additional tests.
• Try to not ask leading questions. It’s very hard to avoid this… but don’t sway the results by leading someone down a certain response path.
• Session recordings are killer. Make sure you watch each video very carefully. I’ve found you can pick up some interesting and important things while watching and listening to users that are trying to accomplish a task (or just while they are reviewing a site).
• Take a lot of notes… I had a text editor up and running so I could timestamp important points in the videos. Then it was easy to go back to those clips later on while compiling my results.
• Try to gain both quantitative and qualitative feedback from users. Sure, multiple choice questions are great and can be quick and easy, but open-ended questions can yield important findings that might not be top-of-mind when crafting your test. And then layer on videos of each session, and you can gain a solid view of how real users view your site, content, and writers.
• Find the right balance for the number of participants. Usertesting.com recommends up to 15 participants for a test. Don’t overload your test, which can lead to data overkill. Try different numbers of participants over a series of tests to see what yields the most valuable results. For some tests, 5 participants might be enough, while other tests might require 15 (or more).  

Summary – User testing can be a powerful tool for sites impacted by Google’s core ranking updates
Google has explained many times that it is looking at many factors when it comes to broad core ranking updates. That includes content-quality, technical SEO, user experience (UX), advertising situation, E-A-T, and more. Google’s John Mueller has also explained that it’s important to take a step back and objectively analyze your site.

Well, a great way to objectively analyze your site is by conducting user testing. Then you can have objective third-parties go through your site, content, features, etc., and provide real feedback. I’ve found this process to be extremely valuable when helping companies impacted by major algorithm updates since it can surface qualitative feedback that is hard to receive via other means. I recommend trying this out for your own site (even if you haven’t been impacted by core updates). I think you’ll dig the results. Good luck.

GG