The Internet Marketing Driver

Percent Human: A list of tools for detecting lower-quality AI content

By Leave a Comment

AI content

As I’ve been sharing examples of sites getting pummeled by the Helpful Content Update (HCU) or the October Spam Update, I’ve also been sharing screenshots from tools that detect AI content (since some sites getting hit are using AI to pump out a lot of lower-quality content – among other things they were doing that could get them in trouble). And based on those screenshots, many people have been asking me which tools I’m using.

So, instead of answering that question a million times (seriously, it might be a million), I figured I would write a quick post listing the top tools I have come across. Then I can just quickly point people to this post versus answering the question over and over.

And note, I’m not saying these tools are foolproof. I have just found them to be pretty darn good at detecting lower-quality AI content. And that’s what we should be trying to detect by the way (not all AI content… but just low-quality AI content that could potentially get a site in trouble SEO-wise).

For example, here is high-quality human content run through a tool:

Detecting human content

And here is an example of lower-quality AI content run through a tool:

Detecting lower quality AI content

Again, it’s not foolproof, but can give you a quick feel for if AI was used to generate the content. Below, I’ll cover my favorite AI content detectors I’ve come across so far. I’ll also keep adding to this list so feel free to ping me on Twitter if you have a tool that’s great at detecting lower-quality AI content!

Here is a list of tools covered in this post for detecting AI content:

1. Writer’s AI content detector tool:
The first tool I’ll cover is from a company that has an AI writing platform (sort of ironic, but does make sense). Also, it seems like the platform is more for assisting writers from what I can see. You can check out their site for more information about the platform. Well, they also have a nifty AI content detector that works very well. You have probably seen my screenshots from the tool several times on Twitter and LinkedIn. :) Below are some examples.

Here is Writer’s tool detecting higher-quality human content:

Writer's AI content detection tool measuring high quality human content

And here is Writer’s tool detecting lower-quality AI content:

Writer's AI content detection tool measuring lower quality ai content

2. Huggingface GPT-2 Output Detector Demo:
If you’re not familiar with Huggingface, it’s one of the top communities and platforms for machine learning. You can check out their site for more information about what they do. Well, they also have a helpful AI content detector tool. Just paste some text and see what it returns. I have found it to be pretty good for detecting lower-quality AI content. 

For example, here is Huggingface’s tool detecting higher quality human content:

Huggingface's AI content detection tool measuring high quality human content

And here is Huggingface’s tool detecting lower-quality AI content:

Writer's AI content detection tool measuring lower quality ai content

3. Giant Language Model Test Room (GLTR.io)
The third tool I’ll cover was actually down recently, but I had heard good things about it from several people (when it was working). It ends up there was a server issue and the tool was hanging. Well, the GLTR is back online now and I’ve been testing it to see how well it detects AI content.

The tool was developed by Hendrik Strobelt, Sebastian Gerhmann, and Alexander Rush from the MIT-IBM Watson AI Lab and Harvard NLP. It’s definitely not as intuitive as the first tools I covered, but once you get the hang of it, it can definitely be helpful.

How it works:
You can paste text into the tool and view a visual representation of the analysis, along with several histograms providing statistics about the text. I think most people will focus on the visual representation to get a feel for how likely each word would be the predicted word based on the word to its left. And that can help you identify if a text was written by AI or by a human. Again, nothing is foolproof, but it can be helpful (and I’ve found the tool does work well). To learn more about GLTR and how it works, you can read the detailed introduction on the site.

For example, if a word is highlighted in green, it’s in the top 10 of most likely predicted words based on the word to its left. Yellow highlighting indicates it’s in the top 100 predictions, red in the top 1,000, and the rest would be highlighted in purple (even less unlikely to be predicted).

The fraction of red and purple words (unlikely predictions) increases when the text was written by a human. If you see a lot of green and yellow highlighting, then it can indicate the text contains many predicted words based on the language model (signaling the text could have been written by AI).

Here are two examples. The first shows AI content (many words highlighted in green and yellow). This text was generated via GPT-2.

Giant Language Model Test Room (GLTR) analysis of AI generated content.

And here is an example from one of my articles about broad core updates. Notice there are many words highlighted in red, and several purple words as well (signaling this is human-written text).

Giant Language Model Test Room (GLTR) analysis of human-written content.

Summary: Although not foolproof, tools can be helpful for detecting AI content.
Again, I’ve received a ton of questions about which tools I’ve been using to detect lower-quality AI content, so I decided to write this quick post versus answering that question over and over. I hope you find these tools helpful in your own projects. And again, if you know of other tools that I should try out, feel free to ping me on Twitter!

GG

True Destination – Demystifying the confusing, but often accurate, true destination url for redirects in Google Search Console’s coverage reporting

By Leave a Comment

If you are confused when Google reports redirects as other categories, like “blocked by robots.txt”, “soft 404s”, “noindexed”, “404s”, and others, it could be Google silently following the redirect and reporting the status of the true destination url instead. My post covers the situation in detail, and provides examples of this happening in the wild.

While heavily analyzing websites from an SEO standpoint, you will undoubtedly find yourself deep in Google Search Console (GSC) reporting. GSC contains a boatload of data directly from Google and can help site owners and SEOs surface key insights. That said, it’s important to understand the nuances involved with GSC reporting, and how Google determines the information it provides in those reports. Having a clear understanding of what the data is showing is important when taking action to improve SEO.

And there’s no better example of GSC data confusion than the dreaded true destination url for redirects in GSC’s index coverage reporting (and URL inspection tool). I have received so many questions about this from clients that I decided to write this post so I can just point people here versus explaining it again and again.

So, join me on a GSC adventure where we uncover the secrets of the true destination url. Some of you might already know this, but I know some do not. And for those that don’t, this will all make sense very soon. You might not be happy with how this is working, but at least you’ll understand why urls are categorized in certain ways in GSC (and via the URL inspection tool).

What is the dreaded true destination url situation in GSC for redirects?
When viewing the indexing status in GSC of urls that are being redirected, Google reports on the true destination url (even if that url is outside of your own site). For example, if you redirect a url to another url, and that url is not indexable for some reason, GSC will silently follow the redirect and report on the final destination’s status. And that can be super confusing for site owners and SEOs that don’t know this is happening.

Yes, that means you can see urls showing up as “blocked by robots.txt”, “noindexed”, “soft 404”, “404”, and more (when the url you are inspecting is actually redirecting). As you can imagine, many site owners are left confused when they see “blocked by robots.txt” when they know 100% that a url is redirecting.

Google’s John Mueller has been asked about this many times, and he has replied with what I explained above (and does admit it can be a bit confusing). Also, Barry wrote a post covering how this happens with the URL inspection tool based on John’s comments. Even though this has been documented, I find it’s still a very confusing situation for many site owners and SEOs (which is why I’m writing this post).

Here is a tweet of mine with a link to John explaining how Google silently follows redirects (and how that shows up in GSC):

Now that you know this is happening, you might be wondering what this actually looks like in GSC. I’ll cover that next with examples of this happening in the wild.

Examples of Google silently following redirects and reporting the true destination url status in GSC:
Below, I’ll provide examples with screenshots of Google reporting on the true destination urls versus the redirect. Again, this is when the final destination urls are not indexable for some reason.

Blocked by robots.txt:
The url is redirected outside the site to a url that is blocked by robots.txt. Google reports the redirecting url as being “blocked by robots.txt” since the final destination is actually disallowed.

A twist on blocked by robots.txt:
This url redirects first to a tracking url, which is blocked by robots.txt. The final destination is not blocked, but Google can’t follow the first redirect to find the final destination url since it’s disallowed. It just knows that first url in the chain is blocked and reports that in GSC. Below, you can see the second step shows the url is actually blocked by robots.txt (and that’s what is reported in GSC).

Soft 404:
The url redirects to a page that’s a soft 404 (a product is unavailable). Google reports that the redirecting url is a soft 404 (since the true destination url is being seen as a soft 404).

Here is the page the url redirects to (with the product “currently unavailable”). Hence the soft 404:

Noindexed:
Yep, you guessed it. The url redirects to a page that’s noindexed. Google reports the url that is redirecting as noindexed in the coverage reporting:

Crawled, not indexed:
At first glance, you might think the redirect is being reported as “Crawled, not indexed”. Not true! It’s the final destination url that’s not being indexed by Google. Google is reporting “Crawled, not indexed” for the true destination url.

The final destination url is indeed not indexed:

404:
How can Google see a redirect as a 404? It doesn’t. It’s the true destination url that 404s and that’s what is reported in GSC.

404 with domain name change:
This is just a variation on the 404 situation to explain how this works when changing domain names. The url on the old domain redirects to a url on the new domain name, but the url was never migrated (it 404s). So Google reports that the redirecting url is a 404.

Sorry, more confusion with redirects:
When a url redirects to a page that resolves with a 200 header response code, and is indexed, the URL inspection tool reports accurately about the redirect (and says that initial url is a redirect and not indexed), but Google shows the canonical as the true destination url (where the redirect leads to). Talk about confusing, especially based on everything I explained above with the other examples where the redirecting urls are being reported as something different than a redirect…

A possible solution in GSC to clear up the confusion:
So, how can this be more intuitive? I think if GSC actually provided a message that it’s reporting on the true destination url, it could clear up the confusion for site owners and SEOs. Below, I have mocked up what this can look like in GSC. If Daniel Waisberg is reading (and I hope you are), then please add this!

Summary: Clearing up the confusion with redirects and destination url reporting.
I hope this post helped you understand how Google is silently following redirects and reporting on the true destination urls in GSC. I know it’s a confusing topic for many site owners and SEOs and I’m sure it has led to many head-scratching moments. Just keep in mind that as of now, GSC is reporting on the true destination urls when a url redirects. So don’t be surprised when you notice redirects in other categories in GSC’s coverage reporting (or when using the url inspection tool). And who knows, maybe the GSC product team will implement that message I mocked up above…

GG

Google’s September 2022 Broad Core Product Reviews Update (BCPRU) – The complexity and confusion when major algorithm updates overlap

By Leave a Comment

Google September 2022 broad core product reviews update

Well, SEOs and site owners had a heck of an end to the summer of 2022. It all started with the Helpful Content Update (HCU), which rolled out on August 25, 2022. The rollout of Google’s new site-wide signal took a little more than two weeks to complete (and it only seemed to impact the most egregious sites). I covered that heavily on Twitter while analyzing the update.

But we didn’t have much time to rest after the HCU rollout completed, since Google then rolled out the September 2022 broad core update. That was surprising, especially since I spoke with Google about the HCU and Product Reviews Update prior to the HCU rollout, and didn’t hear anything about a broad core update launch that would follow!

So, now we had a broad core update rolling out that can make the earth shake, and after a new site-wide ranking signal rolled out (the HCU). What else could happen?

In a move that surprised many (due to timing), Google then rolled out the September 2022 Product Reviews Update (PRU). And PRUs can be core update-like for sites that contain products reviews content. Although we knew the next PRU was coming, the timing of the rollout was extremely surprising since Google overlapped the broad core update and the Product Reviews Update.

That’s right, it’s another algo sandwich from Google, which can bring a ton of confusion for site owners and SEOs. To make matters even more confusing, both the broad core update and the PRU completed rolling out on the same exact day (September 26, 2022). Needless to say, many are confused about which update impacted their sites.

Ranking updates page for the September 2022 broad core update and Product Reviews Update

And if you’re an SEO history buff, then you might know that algo sandwiches are not new for Google, just rare (pun intended). Google has rolled out overlapping updates, or updates very close to one another, several times in the past. The one that really sticks out to me was the Panda, Penguin, Panda algorithm sandwich from April of 2012 (where all three updates were rolled out within a 10-day period). It was a triple decker sandwich packed with thin content, spammy links, and topped with Google’s famous hot sauce. As you can guess, many were confused about what hit them at the time… which led to several of my posts about Pandeguin (the combination of Panda and Penguin).

Here’s a graph from my post about Pandeguin showing a site impacted by all three updates:

Pandeguin impact

Covering The Confusion Based On Overlapping Major Algorithm Updates:
In this post, I’m going to cover the two overlapping major algorithm updates that rolled out in September 2022, the confusion that set in for some site owners and SEOs, interesting things I saw during the combined rollout, my recommendations for sites impacted during the algo sandwich, and more.

So strap yourself in, grab your favorite sandwich sauce, and maybe a pickle for good luck. It’s going to be a bumpy ride in Gabe’s Deli for this post. Let’s jump in.

First, here’s a quick table of contents for those that want to jump around the post:

Visualizing the confusion. Was it the broad core update or the Product Reviews Update?
First, let’s get the dates right. The broad core update started rolling out on September 12, 2022. And many sites saw impact very quickly with the broad core update. I started documenting movement about one day into the rollout, which is quick.

For example, here are several sites seeing immediate impact:

Early impact from the September 2022 broad core update
More early impact from the September 2022 broad core update
Another example of early impact from the September 2022 broad core update
Final example of early impact from the September 2022 broad core update

Then the Product Reviews Update started rolling out on September 20, 2022 in the middle of the broad core update rollout. It was eight days into the rollout of the broad core update. And this is where things get interesting, and confusing. Many sites saw a ton of movement starting right at that point. Some were product reviews sites, but some were not. Not even close actually…

Here are some examples of product review sites seeing movement when the September PRU rolled out. This would be clear PRU impact in my opinion (without confusion based on the broad core update):

Example of product review site impacted by September 2022 Product Reviews Update
Another example of product review site impacted by September 2022 Product Reviews Update
Third example of product review site impacted by September 2022 Product Reviews Update
Final example of product review site impacted by September 2022 Product Reviews Update

But not all sites impacted were as clear as those… There were many sites that don’t contain reviews that were impacted heavily starting on 9/20 (right when the PRU started rolling out). And that led to massive confusion about which update actually impacted sites seeing a lot of movement. In other words, was it the broad core update or the Product Reviews Update impacting the site? Only Google knows… or do they? I’ll cover more about that soon.

Here are some examples of sites that don’t contain reviews or affiliate content at all that were heavily impacted starting on 9/20. These were reference sites, news sites (without affiliate content), e-commerce sites, recipe sites (without affiliate content), and more.

Non-review site impacted on the same date the Product Reviews Update rolled out.
Another non-review site impacted on the same date the Product Reviews Update rolled out.
Third example of a non-review site impacted on the same date the Product Reviews Update rolled out.
Final example of a non-review site impacted on the same date the Product Reviews Update rolled out.

Welcome To Google Land – The Overlapping of Major Algorithm Updates
I have said “Welcome to Google Land” many times over the years, and for good reason. It can sometimes feel like you are on a roller coaster, run by AI, in a land of confusion, with no games or stuffed animals at the amusement park to make you feel better.

Google usually tries to keep major algorithm updates separate so site owners can better understand which update actually impacted their sites. Google’s Danny Sullivan has explained that in the past (and even right before this algo sandwich rolled out!)

But, and like I said earlier, there are times that major updates do overlap. This was a great example of that… When the September PRU rolled out during the September broad core update, Google provided some advice to site owners. Google’s advice was basically that if you have product reviews, then it’s probably the PRU impacting your site. If you don’t, then it’s probably not.

And as you can guess, the word “probably” definitely caused some concern. And it wasn’t long before we saw sites that didn’t have product reviews that were impacted heavily when the PRU rolled out.

Here are tweets from Google about the overlapping rollout:

Don’t assume it’s the PRU, it could be the broad core update:
It’s also worth noting that since the core update was still rolling out, site owners could not simply assume it was the PRU impacting their sites on 9/20. Based on what I saw across many sites, I do believe we saw an uptick from the broad core update right on 9/20 when the PRU rolled out. That was either coincidental, or not. But again, many sites without product reviews were heavily impacted on that date.

For example, here is visibility trending from a recipe site without any product reviews or affiliate links that was impacted on 9/20. Some users are leaving a quick review of the recipe in the comments, but that’s not really what the PRU should be targeting. So either the PRU is flawed there or it was the broad core update impacting the site. Personally, I believe it was the core update impacting the site, but I can’t say for sure. The site owner is super-confused. They were impacted by a previous broad core update by the way, which is interesting.

Recipe site without reviews or affiliate links impacted on 9/20 when the Sep 2022 product reviews update rolled out

Here is a quote from the site owner about the situation:

“When the September core update started rolling out my site wasn’t affected at all. But the day the PRU started, it took a relatively big hit (it dropped by about 20%). The site doesn’t have product reviews or affiliate content at all. Unfortunately, the overlapping updates can send you down a rabbit hole trying to fix things that aren’t actually a problem… since you don’t have a clear idea of which update caused the drop.

“For example, if it was the Product Reviews Update, then you would think I should be focused on improving reviews content (but I don’t have any reviews)… Or is it the comments that sometimes have quick reviews of recipes from users? And if it was the core update causing the drop, should I focus on improving the site overall? It’s just very confusing when all I’m trying to do is publish great recipes for my users…”

And if you think that’s a tough situation, then look at this trending (if you dare):
The site contains mostly informational content, but does include some reviews (and affiliate links). Google is clearly having major issues understanding the type of site and whether it should rank well. The site has experienced massive swings in rankings during multiple major algorithm updates (including reversals outside of those updates). Below, you can see impact from a broad core update, Product Reviews Updates, and then random swings outside of major updates.

Insane trending for a site impacted by several major algorithm updates.

Which then led to others sharing their trending on Twitter, which showed similar ups and downs.

Note, I’ll come back to this case shortly… since the next section ties in nicely. Yep, it’s like Inception for SEO. :)

A final September PRU tremor correcting some issues with the July PRU:
So, based on what I explained with the 9/20 impact, could there be a flaw with the September PRU? It’s possible… and I was pretty vocal about some problems I saw with the July PRU. At the time, I said that we could see Google correct those flaws via a tremor or with the next PRU.

Here is my tweet from July explaining we might see a correction:

Thankfully, a change was rolled out! At the very end of the rollout of the September PRU (on 9/25), a number of sites surged back from the dead that were impacted by the July PRU. I tweeted several examples when this happened.

Site surging back from the dead during a late PRU tremor
Another site surging back from the dead during a late PRU tremor
A third site surging back from the dead during a late PRU tremor

Time will tell if some issues with the September PRU get corrected and those sites bounce back like these did on 9/25.

And remember that site from earlier with insane trending? Well, here we go again. That site surged back again on 10/15. Yep, after the broad core update and PRU completed. And after the site dropped heavily during the late PRU tremor on 9/25. Again, Google is having a very hard time understanding where this site fits in, the type of content it contains, etc.

More insane trending for a site impacted by multiple major algorithm updates.

More Negative Impact for Sites Hit By Helpful Content Update (HCU)
After the Helpful Content Update rolled out, there was some confusion about if the HCU could contribute to broad core updates (as another signal). Google’s Danny Sullivan explained the updates are separate, but if you were impacted by the HCU, and if you have broad core update issues, then the combined effect might not be optimal for you… Basically, the effect can be compounded.

Here is a tweet from Danny about this:

And here are some sites that dropped with the Helpful Content Update and then saw more of a drop with the September broad core update:

Site impacted by the Helpful Content Update that saw more of a drop with the Sep 2022 broad core update
Another site impacted by the Helpful Content Update that saw more of a drop with the Sep 2022 broad core update
Third site impacted by the Helpful Content Update that saw more of a drop with the Sep 2022 broad core update

Google on how the PRU evaluates sites. Site-level or url-level?
Based on the varying levels of impact by the Product Reviews Update, and how it impacted some sites that contain a mix of informational content and reviews, Google received some questions about how the PRU evaluates content. For example, is it evaluating at the url-level or site-level? And, how does Google determine that a piece of content is a review in the first place (especially if the site has other types of content)?

Google first responded quickly about the latter question and recommended that sites might want to add structured data to clearly signal to Google that the content was a review. Then when questioned about that point, Google took some more time internally before responding.

Then Danny Sullivan responded and first explained more about how the PRU evaluates websites. He explained that if your site contains a lot of product reviews, then the PRU can be like a site-wide evaluation. i.e. All content can be evaluated… But, if your site contains a mix of content, and reviews don’t make up a large portion of the content, then the PRU evaluates more on the url-level.

Here are Danny’s tweets about that:

This made complete sense to me based on analyzing many sites impacted by Product Reviews Updates. For sites where a majority of the content is comprised of product reviews, the site could be heavily impacted by the PRU (and any content on the site could be negatively impacted).

For example, here is a site that focuses on product reviews that was impacted heavily by the April 2021 PRU. Notice the massive drop when the update rolls out:

Extreme drop for a product reviews site during the April 2021 PRU

And for sites that contain a mix of content, like news sites that also publish some product reviews, then it wouldn’t be like a site-wide evaluation. In other words, not all content would be evaluated by the PRU. For example, here is a news site that contains review content in a specific section. The site overall took a hit when the PRU rolled out, but the section with reviews REALLY took a hit.

First, here is the drop overall for the site:

News site also containing reviews dropping with the Sep 2022 PRU

And here is the drop for the reviews section:

News site with reviews section dropping with the Sep 2022 PRU

This does reinforce the idea that adding all reviews to a section could help Google identify where reviews are on the site. John Mueller explained that the PRU works more broadly on a site or section-level depending on how the site is structured content-wise.

And that dovetails nicely into the next section of my post about structured data and Google identifying reviews…

Structured Data helping Google understand if your content contains reviews. Really?
For sites that contain a mix of content, Danny Sullivan explained that Google can use structured to help it understand if a piece of content is a review. But, it’s not required and it’s just one of several signals they use to understand if content is a review.

And this from earlier in the conversation:

I’ve been vocal that this is pretty ridiculous to me. I mean, Google with all of its unbelievable natural language processing power needs site owners to feed it structured data to understand if it’s a review? That’s crazy. Also, many site owners don’t even know what structured data is and how to use it. That said, if your site contains a mix of content (including reviews), then I would add structured data to signal to Google which pieces of content are indeed reviews.

And if you think that structured data statement from Google caused more confusion, you would be right. I had several site owners reach out to me claiming they were hit by the PRU because of structured data errors, the wrong structured data used, etc. Unfortunately, even though I don’t believe that’s the case, I can’t say it’s NOT the case with 100% certainty.

Google adds examples of product reviews to its documentation.
With each Product Reviews Update, the algorithm continues to evolve. One question I get often is about the types of content that can be impacted by the PRU. For example, is it supposed to just target sites with product reviews or can sites with other types of reviews content be impacted? I’ve seen sites with user-generated content (UGC) reviews get impacted in the past and I’m not sure Google is really targeting that type of review with the PRU. Well, I’ve seen less and less of that with each Product Reviews Update. Google does seem to be focusing just on product reviews with the latest iterations of the PRU.

Also, Google recently refined one of the help documents about the Product Reviews Update. Specifically, Google added three examples of the types of reviews content that can be created by site owners (and I’m assuming Google is saying these are the types of review content that can be evaluated by the PRU – at least for now). And those examples don’t contain UGC reviews.

It’s just worth noting for anyone creating reviews content (or any site owner that has UGC reviews).

Google adds examples of product review pages in their documentation for site owners and SEOs.

Final tips and recommendations for site owners impacted during the September 2022 algorithm sandwich:
If you were impacted during the latest algorithm sandwich, then the tips below could help you get moving in the right direction. I hope the following recommendations help cut through some of the confusion:

  • If you were impacted starting on 9/12, and before 9/20, then you were impacted by the broad core update. You can read my posts about broad core updates to learn more about them, how to identify why you were impacted, and learn the best path forward from a remediation standpoint.
  • If you were impacted starting on 9/20 when the PRU rolled out, and you have a mix of content on your site beyond reviews, do not assume it was the PRU. It could have been the broad core update. I covered this earlier in the post, and I have seen many sites impacted on 9/20 that had no reviews or affiliate content at all.
  • If you were impacted starting on 9/20, and you do have a lot of product reviews content, then you should work to improve your content based on Google’s best practices. You can read my previous posts about Product Reviews Updates to learn more about them.
  • If you have a mix of content on your site, including reviews, then I recommend adding structured data to identify the reviews content (since Google explained it uses structured data as one signal for identifying reviews). I still think it’s crazy that site owners need to do this… but I would probably do it to make sure Google can understand what is reviews content and what’s not. Remember, we’re dealing with machine learning systems and mistakes can definitely be made by Google on this front. Just check the reversals in my post above to see examples of that.
  • If impacted by the September broad core update, run a delta report to understand what dropped, and why. For example (if it was the core update), was it a relevancy adjustment, intent shift, or is it overall site quality problems. Then form a remediation plan based on what you find.
  • If impacted by a major algorithm update, don’t just compare specific urls in the SERPs if that’s the case. With site-level quality algorithms at play, the overall quality evaluation could be dragging rankings down. And for some urls, the content on that page might have little to do with that url specifically dropping. Improve overall… that’s what Google wants to see.
  • For broad core update remediation, using a “kitchen sink” approach is your best path in my opinion. That’s where you surface all potential quality problems and work hard to fix as many as you can. Look to improve overall. That’s all you can do in the age of machine learning-based major algorithm updates.

Summary – Overlapping major algorithm updates can cause confusion for sites owners and SEOs. Try your best to cut through that confusion…
I hope this post helped you better understand the major algorithm updates that rolled out in September of 2022. Unfortunately, two of those major updates overlapped for a week. And when that happens, it can cause massive confusion for site owners and SEOs. By reviewing the points in my post, and understanding the dates you were impacted, you can form a plan of attack remediation-wise.

And like I explained in my post, don’t be surprised if we see some corrections and adjustments based on the latest updates. Again, “Welcome To Google Land”. Good luck.

GG

Back to top>>

Google Multisearch – Exploring how “Searching outside the box” is being tracked in Google Search Console (GSC) and Google Analytics (GA)

By Leave a Comment

Google Multisearch Tracking

Google first announced multisearch in April 2022 (right before Search I/O 2022) and it was really the beginning of Lens entering the spotlight. With multisearch, you can use Google Lens to analyze a photo and then add to your search by typing in a query. Then Google will display more results based on the query and photo analysis.

For example, after multisearch was announced I took a photo of a cocktail via Lens and then entered “recipe” via multisearch. Then Google returned recipe results for me based on analyzing the photo.

That’s powerful, but there are challenges. The biggest problem in my opinion is that most people:

  1. Don’t know about Lens at all (or don’t know much about it).
  2. Have no idea how to use Lens.

Then at Search I/O 2022 in May, Google announced Multisearch near me, which will enable you to use Lens to snap a photo, enter “near me” via multisearch, and then Google will help you find local businesses that can provide that product. The example they provided was taking a photo of a dish you enjoyed and finding local businesses that offer that dish. This should be rolling out in the fall of 2022 (which was just announced at Search On 2022 last week).

But with Search ON 2022, there was more focus on Lens and multisearch, pun intended. First, multisearch will be expanded beyond just English to 70 languages in the coming months. Also, Google announced a shop option at the top of the  SERPs based on using Lens to analyze photos of products. That will provide more opportunities for people to use Lens, and then multisearch, to shop for products based on the photos they take. So, Google can jumpstart your shopping experience with this new addition.

Tracking Multisearch: It’s complex.
Multisearch is cool and exciting, but as SEOs and site owners, how do we know if people are finding our content via Lens and multisearch? That’s a question that Jamie Indigo asked on Twitter earlier in the week. It’s a great question and I dug in a bit to see how it was working and being tracked.

After Jamie’s tweet, I ended up running several tests to see how multisearch was being tracked in Google Search Console (GSC) and in Google Analytics (GA). Below, I’ll provide the results of my testing to see how multisearch is being tracked, or if it’s being tracked at all.

The Lens and Multisearch SERP: “Visual Matches” versus Google Images.
After using Lens and multisearch, you are presented with a SERP that resembles Google Images, but it’s not really Google Images. Instead, it’s labeled “Visual Matches” and I believe it’s a special treatment based on using Lens. It even says, “Results for this feature are still experimental.”

Google multisearch and recipes

From that SERP, you can tap a listing which takes you to a Google Images-like SERP where you can view the photo and tap through to the website if you want. But again, it’s not Google Images (at least for now).

Google multisearch images

So, how is this tracked in Google Search Console (GSC)? You might think the impressions and clicks will show up in the Performance reporting for Image Search. That would make the most sense at this point (until Google provides specific reporting for Lens and multisearch – if they do that at some point).

Well, I wanted to test this out so I ran three tests across clients to see if the GSC reporting would show the activity. I used Lens and multisearch using specific queries while documenting the landing pages. Then I waited for data to arrive in GSC. I’ll cover the results below.

Google Search Console Tracking for Multisearch:
Well, this will be quick. I’m sorry to say that there is currently no tracking at this point in Search Console for multisearch. Again, you would think impressions and clicks would show up in the Performance reporting for Google Images, but nothing shows up based on my testing. I checked both Web Search and Image Search and the activity was not tracked. So the “Visual Matches” SERP isn’t being tracked like Google Images.

Google search console tracking for multisearch (image search)
Google search console tracking for multisearch (web search)

Google Analytics Tracking for Multisearch:
What about visits from multisearch? How do they show up in Google Analytics? Well, I see visits showing up as either Direct Traffic or via Google quicksearchbox (like Discover can shows up as sometimes). So, you basically can’t track multisearch specifically in GA either. Multisearch will be mixed into Direct Traffic or Google quicksearchbox (which again, is what some Discover traffic shows up as).

Multisearch showing up as “GoogleQuickSearchBox” in log files:

Multisearch showing up as GoogleQuickSearchBox in log files

Multisearch showing up as Direct Traffic in GA:

Multisearch visits showing up as Direct Traffic in Google Analytics

How multisearch tracking could look in GSC: A mockup.
I pinged Daniel Waisberg from Google who leads the GSC product team about the tracking issue, along with a mockup of what the reporting could look like. There are several options, but I think the most logical would be a filter for multisearch in the Performance reporting (sitting alongside Web, Image, Video, and News).

Here is the mockup I sent Daniel:

Google Search Console mockup of multisearch reporting.

Summary: Multisearch is powerful, but site owners need reporting to understand usage!
Again, I think multisearch is super-powerful. The ability to search via Lens and then add a query to the analysis to find more information can help on multiple levels (including local, shopping, and more). But site owners need to know how their audience is using multisearch to find their content. And that means tracking in GSC.

We are very early on with multisearch, so I’m hopeful we will see some type of tracking show up in Search Console. But I’m also aware that there needs to be significant usage before that happens. I can’t see Google releasing reporting in GSC that shows little activity for a majority of sites.

Time will tell when multisearch reporting shows up. Until then, you should definitely check out and test Lens and multisearch. And after using it, spread the word. It’s a powerful way to “Search outside the box”, as Google would say.

GG

Sitebulb Server – Technical Tips And Tricks For Setting Up A Powerful DIY Enterprise Crawler (On A Budget)

By Leave a Comment

Sitebulb Server

When performing SEO audits, crawling is ultra-important. For SEOs and site owners, there are several options available from local crawlers to enterprise crawlers (SAAS services). I’ve been extremely vocal over the years about my favorite crawling tools, which are Screaming Frog, Sitebulb, DeepCrawl, and more recently, JetOctopus. Screaming Frog and Sitebulb are typically local crawlers, while DeepCrawl and JetOctopus are excellent and powerful enterprise crawlers.

When referring to Screaming Frog and Sitebulb, I said “typically local crawlers” since there are ways to hack a local crawler into a SAAS crawler. For example, I have spun up several AWS servers, installed Screaming Frog and Sitebulb on them, and then I can crawl remotely. The core benefit being that I free up my local resources to focus on other things while my AWS servers do the heavy lifting crawling-wise.

That has worked pretty well, but there’s a new solution for you Do-It-Yourself’ers. It’s called Sitebulb Server, and it’s currently in beta. I’ve been using it for several months and wanted to cover some tips and tricks in a blog post. I think it’s a powerful solution that can take you from local to enterprise on a budget.

Note, I won’t be covering everything you need to know in this post. Instead, I wanted to cover how it works, some technical tips and tricks, and some watchouts. I’m sure the team over at Sitebulb can answer any other questions you have (they have been super helpful over the years and with this beta). And you can always ping me on Twitter if you run into any issues. If I can answer those questions quickly, I will.

What is Sitebulb Server Exactly?
Sitebulb Server is a way for you to set up a special version of Sitebulb on a separate server, which can run crawls while not bogging down your local resources. With the standard version of Sitebulb, most users run it on their local computers. That’s fine, but it can definitely bog down your system and take up bandwidth. With Sitebulb Server, that all happens on a separate server. Then you can use a special version of Sitebulb on your desktop to connect to your server. And when you do that, you can access the audits like you had run them on your local machine. It’s awesome to be able to do that.

I mentioned earlier that you could always set up a separate remote server and run Sitebulb (or Screaming Frog). I have done this for years and it works pretty well (although you couldn’t run multiple crawls at the same time). Well, Sitebulb Server is a remote crawling server, but on steroids. It’s built to run multiple crawls at the same time while enabling you to connect to any of those crawls from your own desktop app. In addition, multiple team members can access those crawls from Sitebulb Server. So if you have a team of SEOs working on an audit, then Sitebulb Server can be a strong DIY solution for accessing crawl data across those team members.

The ability to crawl sites concurrently on a remote server is amazing:

Crawl multiple sites using Sitebulb Server

You can access your server from anywhere in order to audit the crawl data like it was sitting on your local machine:

Access crawl data from anywhere via Sitebulb Server

The Biggest Obstacle IMO – The scary, confusing, cryptic, but often easy, server setup.
This all sounds great, right? But what’s the biggest obstacle or hoop that you need to jump through? Undoubtedly, it’s the server setup. I ran into this when first setting up AWS instances to run their own versions of Screaming Frog and Sitebulb. It’s a cryptic process that many SEOs and site owners aren’t familiar with. It’s not necessarily hard, but definitely an obstacle in my opinion. I find many SEOs have not set up separate servers for crawling and I know a number that ran into snags while trying to set them up.

Well, Sitebulb to the rescue. Patrick and Gareth from Sitebulb have created excellent documentation for setting up Sitebulb Server, how to set up remote servers (including AWS and Google Cloud Compute), and more. You can read more in their help documentation, which also includes video clips (which are amazing when you are trying to set up remote servers). Sometimes a picture is worth a thousand words.

For example, here is a video clip Sitebulb put together for setting up Sitebulb Server via AWS:

Note, I personally use AWS, and that has worked well, but you can use whatever setup you want. You can use a dedicated server, AWS, Google Cloud Compute, a spare computer on your local network, etc. Once you set up a server, which typically doesn’t take long, then you can move forward with setting up Sitebulb Server and the special desktop version of Sitebulb that connects to your server.

Disk space and vCPUs: Some important points about your server.
When setting up your server, then it’s important to make sure you have enough disk space and enough vCPUs (or virtual CPUs). They impact how much crawl data you can store and how many threads you can use when crawling.

First, crawls take up a lot of space. And enterprise crawls take up a ton of space. Make sure you select enough disk space based on the types of crawls you typically run. Below is a screenshot from AWS for configuring storage.

Configuring disk storage when setting up Sitebulb Server on AWS

Next up is vCPUs (or virtual CPUs). It’s important to understand that each vCPU is a thread. So if your crawl will take up 5 threads, then you’ll need 5 vCPUs. In addition, when you connect to the server, you are also taking up a thread. And if you want to run multiple crawls at the same time, you need to take that into account as well (even more threads). Below, you can see the AWS instance has 8 vCPUs (or 8 threads for Sitebulb Server).

Selecting the number of vCPUs when setting up Sitebulb Server on AWS

For example, if you run two crawls using 5 threads each, and you are connecting to the server, then you’ll need 11 threads (5 + 5 + 1). I had some questions about this, and Patrick was awesome with getting back to me with more information. The team over at Sitebulb has a wealth of knowledge and they are incredible with helping customers. So, first check their documentation. If you still don’t have an answer, I’m sure they can help you figure out the best solution.

Notes about running crawls concurrently versus queuing them.
Another point of confusion is about running concurrent crawls. In other words, this is how you will run multiple crawls at the same time. This is something typically only reserved for enterprise crawlers, but you can do this now via Sitebulb Server.

First, when setting up your server make sure you check the option for running concurrent crawls. That’s in the server settings section.

Checking concurrent audits in Sitebulb Server

Next, make sure you have the right setting for “Concurrent queue type”. That should be set to “Next based on available threads” and not “First in, first out”. If you have it set to “First in, first out”, then each crawl will run separately (and in order). By using “Concurrent queue type”, the crawls can run at the same time as long as there are enough threads (see my comments earlier about that).

Setting concurrent queue type in Sitebulb Server

And for “Reserved threads”, the number you set is based on the number of team members accessing the server at the same time. If you’re a solo consultant, then you can just set one. If you have two other teammates that will be accessing the server at the same time, then you should have that set to three (you and two teammates).

Setting reserved threads in Sitebulb Server

IP Address Changes When You Stop and Restart AWS
Another confusing topic is related to IP addresses and your AWS instances. Since you are paying when the server is in use, you will typically want to stop that instance when it’s not in use. If not, your costs can start to skyrocket. But here’s the rub. When you stop and the restart your AWS instance, the server gets a new IP address. And that IP address is what you use when connecting your Sitebulb desktop app to your Sitebulb Server. It’s also what you use when connecting to that server via Remote Desktop (for managing the server remotely).

Therefore, you will need to quickly go into to your settings on Sitebulb desktop and change the IP address for your server. It doesn’t take long, it’s not hard to do, but it can cause confusion if you don’t know you have to do that. You basically won’t be able to connect to your Sitebulb Server unless the correct IP address is used.

Changing IP address after stopping and restarting an AWS server

And also remember you will need to change that IP address when connecting via Remote Desktop. If not, your connection will fail. You use Remote Desktop to manage your server remotely (like installing software).

Adding a new IP address via Remote Desktop

Connect to multiple Sitebulb servers from one desktop Sitebulb setup.
Another cool feature of Sitebulb Server is that you can connect to multiple servers from one desktop setup. So, if you need multiple Sitebulb Servers since you need to run many crawls at the same time, you can do that. Just spin up multiple AWS servers or dedicated servers, set up Sitebulb Server on them, and then connect to those servers from your desktop app. Sitebulb Server is extremely scalable on that front.

Add multiple servers in Sitebulb Server
Registering a new server in Sitebulb Server

Important: Open up a network port on your server.
OK, I ran into this issue when setting up Sitebulb Server, so I’m sure others will too. Sitebulb also has this in their documentation, so hopefully you won’t miss it when setting up your own server. But, I wanted to cover it here anyway, since it’s important.

You will probably need to open a network port on your server firewall in order to properly run Sitebulb Server. Network ports are typically closed by default, so you’ll need to create a firewall policy to open port 10401 on your server. It’s easy to do once you know where to go and how to do it, but I think many could miss setting it up. Sitebulb’s video tutorials cover this step in detail, so I won’t recreate the wheel here. But again, it’s important to do.

Opening a network port when setting up Sitebulb Server via AWS

Sitebulb Server – A strong option for running enterprise crawls without bogging down your local setup.
Again, I didn’t want to try and cover everything about Sitebulb Server in this post. Instead, I wanted to cover some technical tips and tricks that SEOs and site owners might run into while setting up and running Sitebulb Server (based on using Sitebulb Server over the past several months). Personally, I have found Sitebulb Server to be a strong solution for running enterprise crawls on a budget. And I think you will too. I recommend reaching out to Patrick and Gareth at Sitebulb to learn more about the options available for trying out Sitebulb Server.

GG