The Internet Marketing Driver

How To Find Lower-Quality Content Being Excluded From Indexing Using Bing’s XML Sitemap Coverage Report (and Its “Content Quality” Flag)

By Leave a Comment

Finding lower-quality content via Bing's Sitemap Index Coverage Report in Bing Webmaster Tools


Bing finally rolled out its XML Sitemap Coverage Report in Bing Webmaster Tools, which is a great addition for site owners. Using the report, you can check indexing levels based on the urls being submitted via XML sitemaps. This is similar to what Google offers in its Coverage reporting, but it’s great to have another major search engine provide this data.

Hello “Content Quality” flag:
When I first dug into the reporting, I quickly started checking urls excluded from indexing across sites. Like Google, Bing provides a number of categories for urls being excluded,  including noindexed, redirected, 404s, and more. But one of those categories really struck me – “Content Quality”. With “Quality” being the most important thing that site owners should focus on, understanding when a major search engine believes you have quality problems, and surfacing those specific urls, is pretty awesome.

Bing's "Content Quality" flag in the sitemap index coverage report


And once you click the “Content quality” category, you can view all of the urls from that sitemap that were flagged as having content quality issues:

Viewing urls flagged with "content quality" issues in Bing Webmaster Tools


Bing is not Google, but Bing is a major search engine: And will Google follow?
With major algorithm updates evaluating quality on several levels, having this information from Bing could potentially help site owners surface and improve lower-quality content. And with Google’s broad core updates, reviews updates, and now helpful content updates, digging into urls flagged as lower quality could help jumpstart a site owner’s analysis. Sure, Bing is not Google, but the content that Bing is surfacing in its Sitemap Index Coverage reporting could be a proxy for what Google also believes is lower-quality content. You don’t want to take that at face value, but it’s definitely worth investigating…

And maybe a bigger question is… will Google follow Bing here and provide a “Content Quality” category in its own Coverage reporting? I know Google has toyed with this idea in the past, but never officially rolled out a content quality category in Search Console. To be honest, I’m not sure that would ever happen, since it could reveal a bit too much of the secret sauce. I know they don’t want to provide too much link data either based on that happening.

I mean, imagine waking up one day and seeing this in Google Search Console. :)


Finding the Index Coverage reporting in Bing Webmaster Tools:
If you have at least 10K urls indexed in Bing, then you should be able to see the index coverage reporting for your site in the Sitemaps reporting. But, based on what I’m seeing, a number of sites do not have that option. If you don’t see the option, then I would make sure you are submitting xml sitemaps in BWT or including a reference to them in your robots.txt file.

For example, here is a large-scale site with sitemaps in BWT, but the index coverage option isn’t available.


Maybe the Index Coverage reporting is still rolling out to more sites… I’ll reach out to Bing’s Fabrice Canel to see why those sites don’t have index coverage reporting and then update this post with more information.

Reviewing content quality problems across sites: Were the urls actually low quality?
I was eager to investigate the “Content Quality” category across sites to see what types of content were surfaced there. So I dug in across several sites, and across several verticals. I’ll quickly cover what I found below.

First, although many of the urls were ones that I would consider lower-quality or thin, not all were. Do not take Bing’s word blindly… you definitely need to review the urls yourself. Some were exactly what I would surface as lower-quality, while others seemed ok for users (they were not great, but not terrible either)…

For example, I found the following types of lower-quality urls in the reporting across sites:

  • Short and unhelpful Q&A posts.
  • Thin press releases.
  • Thinner and dated news articles.
  • Spider traps. Little content leading to more thinner pages.
  • Ultra-thin business or organization listing pages.
  • Lower-quality content focused on sensitive categories (YMYL).
  • Thin video pages covered in ads.
  • Low-quality “reference” content.
  • Thin user profile pages.
  • Thin tutorials.

And more…

More Ways To Find Content Quality Problems in BWT:
After tweeting about this the other day, and thanking Fabrice Canel from Bing, he replied with an interesting note. Fabrice explained that the Index Coverage reporting wasn’t the only place you can surface content quality problems in Bing Webmaster Tools. He explained you can also see this when inspecting specific urls and via Site Explorer.

When checking the link he provided, I noticed that Site Explorer was filtered by “URLs with other issues”. So it seems that category means the same thing as “Content Quality” in the Index Coverage reporting for sitemaps. In other words, it won’t say “Content Quality” in Site Explorer, but it means the same thing.

Finding quality problems in the Site Explorer feature in Bing Webmaster Tools


And when inspecting specific urls that were flagged as lower quality in the Sitemap Index Coverage reporting, I typically saw other categories appear for why the urls weren’t indexed. It did not say “Content Quality”. Fabrice did say the data might not be in sync and there could be a 1-2 day lag there between the tools, but it’s worth noting.

For example, a url that was flagged as “Content quality” in the Sitemap Index Coverage reporting actually yielded “Discovered but not crawled” when inspecting that url. That category can signal quality problems too, but it doesn’t say “Content quality”.

Cross-referencing the url inspection tool in Bing Webmaster Tools for urls that are flagged as low quality.


Summary – “Content Quality” is being flagged by a major search engine. Dig in there…
Again, I was pretty excited to see Bing Webmaster Tools provide a flag for content quality. With so much emphasis on “quality” from the major search engines, it’s great to dig in and analyze urls being surfaced as having quality issues. The reporting will never be perfect, and I would not blindly act on what’s being surfaced there, but it’s a nice starting point for site owners trying to understand content quality issues across their sites. I highly recommend digging in there. :)

GG

Bing Chat in the Edge Sidebar – An AI companion that can summarize articles, provide additional information, and even generate new content as you browse the web

By Leave a Comment

I’ve been heavily testing Bing AI Chat since last week and it’s been fascinating to see how it works both in the search results and in chat mode. As I explained in my post about the “code red” that triggered thousands of “code reds” at publishers, AI in the SERPs is one of the most important, and monumental changes, we have ever seen in the search industry. And it can potentially have a huge impact on publishers across the web. The financial impact could be massive.

During Microsoft’s presentation in Seattle, they showed off Bing Chat in the Edge sidebar, but I hadn’t seen it live during my testing… until yesterday! This is where Bing Chat can act as an AI companion while you browse the web (sitting in the Edge sidebar). For example, you can use Bing Chat to answer questions based on the content you are viewing in a tab you have open in Edge. And that opens up a ton of opportunities to have Bing AI help you as you browse the web. For example, you can ask it to summarize the article you are reading, pick out the most important information, understand who the author is, and more.

Well, yesterday I took one more look at the sidebar in Edge to see if Bing Chat would show up, and I’m glad I did! I quickly noticed I had access to Bing Chat in the Edge sidebar! I shared a Twitter thread with some findings based on my testing, but I wanted to document those tests here on my blog.

Below I’ve included several screenshots of using Bing Chat in the Edge sidebar. Again, it’s like an AI companion as you browse the web. I’ll keep testing and adding to the this post.

Permission to access browser web content from the Edge sidebar:
First, Bing Chat in the Edge sidebar asked if it can access browser web content to “get the best experience with relevant responses”. This is so you can select text for highlighting assistance, generate page summaries, and more.

Bing Chat asking for permission to access browser web content.

Summarizing an article:

Bing Chat in the Edge sidebar summarizing a blog post.

As you can see, although I asked Bing Chat to summarize my article, it also pulled in information based on other sites that were writing about the overall topic. The good news is that my site was the core citation, however, there were several other citations listed there, including Bing citing itself (which I have reported before). But that Bing citation led to a search result with my blog post winning the featured snippet. So, I’m cool with that. :)

Bing Chat in the Edge sidebar citing Bing and leading to a featured snippet.

Understanding more about the author of an article or blog post:
Based on pulling up some articles from some top people in the SEO industry, and our interest in Google and Bing understanding more about authors of content, I decided to ask Bing Chat who the author was based on the content I was reading in Edge. Bing Chat did not do very well… at all. You can see it doesn’t know Lily Ray wrote the post and it believes a journalist from The Atlantic wrote Barry Schwartz’s post.

Bing Chat in the Edge sidebar identifying the author of an article.
Another example of Bing Chat in the Edge sidebar identifying the author of an article.

Although, I have seen Bing Chat in the Edge sidebar sometimes respond with the correct author information. For example, when I asked Bing Chat in the Edge sidebar who wrote my blog post. It responded correctly:

I tweeted about the author situation on Saturday and it was great to see Fabrice Canel from Microsoft say that he passed that feedback along to the Bing Chat team.

https://twitter.com/facan/status/1627109124919140353

Selecting text and taking action (like code):
In the Bing presentation in Seattle, Microsoft demonstrated selecting code on a page and asking Bing Chat to refine it. So I searched for a PHP code example, highlighted it, and Bing Chat asked me what I want to do with that text. I could ask Bing Chat to explain it, revise, it, expand it, or ignore it. When I chose “Revise”, Bing Chat responded that it needed to move on… so that functionality isn’t active yet. But good to know that’s coming soon.

Selecting text on a webpage and having Bing Chat explain, revise, expand, or ignore it.

Room for ads in the Edge sidebar (and more content modules):
When using Bing Chat proper (not in the Edge sidebar), there is plenty of room for ads in the citations and below the answer. For example, ads typically have shown above the citations when hovering over parts of the answer and additional modules of content can show up under the answer. The modules can be shopping results, news articles, local listings, etc. That also opens up a ton of opportunity for ads to show up while working with Bing Chat.

Well, there is obviously less room in the Edge sidebar, but there still are opportunities for ads and content modules. For example, when I was asked Bing Chat in the Edge sidebar to summarize an article about the Samsung Galaxy S23, it provided shopping results under the response. And clicking that launched a fresh SERP for the query, “Samsung Galaxy S23”. If you focus on ads, keep an eye on what Bing is doing in the sidebar.

Bing Chat in the Edge sidebar providing a shopping module under a response about a product.

Collapsing citations behind a “Click to view” button. Oh no…
In my post covering the potential impact of AI chat to publishers, I explained how Microsoft vowed to continue to link downstream to publishers. Satay Nadella even said they shouldn’t be crawling websites unless they link to them. That was amazing to hear, but I noticed something in the Edge sidebar that was concerning. For some queries, Bing Chat did provide citations in the response, but then collapsed the bottom citations behind a “Click to view” button. That essentially hid the websites that were being cited (which makes it harder to click those links to visit the sites). I’ll provide feedback to Microsoft about this, but it’s clearly something all publishers should watch closely.

Bing Chat in the Edge sidebar collapsing citations behind a "click to view" button.

Using “Compose Mode” in the Edge sidebar to generate content:
In the Edge sidebar, there are three tabs. One is labeled “Chat”, which I’ve been explaining in this post, but the second tab is labeled “Compose”. I dug in a bit and that’s where you can have Bing Chat generate content. You can tell it the style you want, the format, the length, and then have Bing Chat generate the content for you.

One you generate the content, you can copy it to use elsewhere, or you could click a button labeled “Add to site”. That was very interesting to see… so maybe Edge will create content for you and then let you quickly add that content to the page you are viewing? That button doesn’t work yet, so I’ll keep an eye on it. But again, very interesting to see that.

Using Bing Chat in the Edge sidebar to enter compose mode to generate content.

Here is a short blog post I asked Bing Chat in the Edge sidebar to write about how to throw a knuckleball:

Bing Chat in the Edge sidebar writing a short blog post via Compose mode.

And just for the heck of it, I ran that content through a few AI content detection tools like GPTZeroX. Yep, it sees the content as AI-generated. Just a a quick warning if you are going to use Bing Chat for publishing content.

AI content detection tools identifying a shot blog post written by Bing Chat in the Edge sidebar as AI-generated.

Tracking Bing Chat traffic in Google Analytics:
I noticed a new referral in Google Analytics the other day, which was edgeservices.bing.com. I also saw traffic started arriving via that referral right after Bing Chat was announced in early February. After testing visits from both Bing Chat and Bing Chat in the Edge sidebar, I realized that new referral was traffic from Bing Chat in the Edge sidebar. Traffic from Bing Chat proper shows up as Bing Organic (so there is a distinction in Google Analytics). Also, Bing’s Fabrice Canel confirmed to me on Twitter that the traffic was in fact from Bing Chat in the Edge sidebar.

That’s interesting, since Fabrice Canel also announced that Bing Chat data would be available in Bing Webmaster Tools at some point. But now we also have the ability to track sidebar traffic in GA (via the new referral). Just keep that in mind when analyzing traffic in GA.

For example, here is traffic from Bing Chat in the Edge sidebar showing up in GA:

Bing Chat in the Edge sidebar opens key opportunities for Microsoft (and You)
As you can see based on the examples I provided above, having an AI assistant browsing the web with you, that’s ready to answer questions about the content you are viewing, is almost like having a rudimentary version of Jarvis at your side. I covered how Jarvis is where AI chat in the search results is heading, and this is an interesting first step. Now back to my testing… I’ll try to update this post with more examples soon.

GG

The Google “Code Red” That Triggered Thousands of “Code Reds” at Publishers: Bard, Bing Chat, And The Potential Impact of AI in the Search Results

By Leave a Comment

In this post I cover if AI will cite, or not cite, sources in the search results, and the potential impact to publishers that rely on that traffic for revenue. i.e. Will clicks from the SERPs get pummeled like we think it might?

Last week, both Google and Bing announced AI chat in the search results following the incredible rise of OpenAI’s ChatGPT in the fall of 2022. And to be crystal clear, AI chat in Search is one of the most important, and monumental, innovations we have ever seen in the industry (especially with how it can potentially impact the entire web ecosystem). On the one hand, it’s amazing to live through this change. On the other hand, it’s been extremely scary for site owners, publishers, SEOs, and basically anyone that publishes content on the web. With both Google and Bing adding AI chat to the search results, there could be a massive drop in referring traffic from both engines. And that traffic is the lifeblood of many websites and publishers.

The rise of OpenAI and ChatGPT was so fast, and so powerful, that Google issued a “Code Red” internally to address the situation. Even Google founders Larry Page and Sergey Brin were called back in to review Google’s AI projects. And now that “code red” has triggered thousands of more “code reds” at publishers as many try to wrap their heads around Search without attribution (which would mean a lack of, or decrease in, links to publishers from search engines).

Without precious visitors from Search (which often accounts for a majority of a site’s traffic), site owners could have a heck of a time surviving financially. For example, if you run advertising on your site, you need eyeballs to earn revenue. If you run an affiliate site, you need visitors to read your content and click through to e-commerce sites to convert. If you run a news site and are trying to gain subscribers, then you need people visiting your content so they can understand the value you are providing. So on and so forth.

For example, it’s not unusual to see Google organic dominate referring traffic sources for many publishers. And it’s also not unusual to see organic search dominate as a channel. See an example below:

This is why AI chat can be extremely scary for site owners. If AI can provide the answer, and not cite sources, then those visits go away. And even if the engines cite sources at the bottom of an answer (more on that soon), then sites will undoubtedly receive less traffic. So that’s still not a great situation. I have heard from many site owners, from some of the largest publishers on the web, since last week’s announcements, and many are nervous, frustrated, and scared about what’s coming. And it’s coming fast… really fast. ChatGPT launched in November of 2022 and here we are in February of 2023 talking about AI chat rolling out in the search results. The pace of AI change has been mind-blowing.  

In this post, I’ll cover more about AI chat launching in the search results, including Google Bard and Bing AI Chat, how AI will cite sources (or not), how some lesser-known search engines have been handling the AI chat situation for a while, I’ll cover real publisher search data so you can understand the true risk involved, and then I’ll end with the user adoption issue, and where I think this is ultimately going.

So grab a cup of coffee, a Red Bull, or your favorite cocktail, this is going to be a heck of a ride.

AI Chat Rising (or Exploding) and The Dilemma of Citing Sources.
Google was first up with its announcement of Bard on Monday, February 6. And after that blog post launched, with gifs and screenshots of Bard in action, there was a collective mind explosion for SEOs and publishers. The reason? Google’s AI chat was not citing sources in chat mode. i.e. It was not providing links to sites where Bard “learned” about the content, or where you can find more information about the topic. So, Bard was trained on a ton of data, including content from across the web, but it was simply providing answers to questions without citing those sources and driving users downstream.

Here is what Bard looked like in its experimental phase:

See my points from earlier about the revenue implications of losing traffic to AI chat. I was extremely vocal about this as soon as I saw Bard in action… I called it an act of war by Google against publishers. Harsh words? Yep, but I have helped too many sites and publishers over the years and fully understand the negative impact this could have on their businesses. It could be a huge blow to them financially.

Now, Bard was actually providing a carousel of sites below its AI answers when Google showed an example of Bard in the search results. But, the carousel was below the AI answer with a “Read more” label. Note, with carousels, several of the listings aren’t even visible until you scroll through them. So this is better than Bard providing no citations, but it’s still not great for publishers in my opinion.

After the announcement, Google caught tons of flak for not linking to publishers, so I believe we could see citations when Bard rolls out over the next few weeks. Time will tell, but for now the situation is extremely scary for many site owners. It’s also worth noting that Google has not answered any questions about citing sources. And I mean literally nothing has come from anyone at Google about linking to publishers (which makes me think they were unprepared for the question). That’s also scary… so we’ll see where this ends up.

Next up, Bing AI Chat: ChatGPT in the SERPs
Then on Tuesday, Microsoft had a huge event in Seattle (which was kept secret until the night before). As expected, they announced AI chat in the Bing search results, and in my opinion, they did a fantastic job with the event, demos, and implementation. Bing really crushed Google with their presentation, the way AI chat is incorporated into the SERPs, and how it works with Edge (think Chrome for Microsoft). Remember, Microsoft has invested billions of dollars in OpenAI, the creator of ChatGPT, so it’s no surprise that their implementation of AI chat in the SERPs was slick and well thought out.

Yep, Bing AI Chat Contains Citations!
After Bard’s lack of links to publishers, everyone watching was eagerly waiting to see how Bing’s AI chat powered by ChatGPT would handle attribution. And low and behold, Bing’s AI chat cites sources. Yep, they are providing the sites where it “learned” about the answers it’s providing and where you can find more information based on the answers you are receiving from Bing Chat. You can hover over a segment of the answer, see the site where you can learn more about that topic, and yes, even link downstream to the page for more information. Precious search traffic remains!

And then Microsoft CEO Satya Nadella gave a great interview with Nilay Patel from the Verge and reiterated how important it is to Bing to drive traffic downstream to publishers. He explained search is about fair use and they fully intend to provide links to publishers from AI chat. It was amazing to hear this from him… You can watch the video below starting at 6:43:

He even said they won’t crawl publisher websites unless they can link downstream. That’s a bold statement in a new AI world, and publishers rejoiced. Well, sort of. I’ll cover why next.

Bing AI Chat citations are amazing, but Google is exponentially more important for publishers (at least as of now):
Again, hearing Satya Nadella’s comments about linking downstream to publishers was great, but the truth of the matter is that Bing just doesn’t drive a ton of traffic for most publishers. Google typically drives exponentially more traffic than Bing. I have helped many sites over the years and have seen the massive difference in referring traffic time and time again.

For example, here is a comparison of clicks and impressions data for a site (comparing Google and Bing). This data is directly from each search engine:

For the first site, Google drove 13.2M clicks over the past three months:

And the same site saw just 404K clicks from Bing. Note, that increase you see is due to a major algorithm update that Bing confirmed (where they used the new AI model on Bing’s core ranking engine):

The next site saw 3.8M clicks from Google over the past 3 months:

But only 52.9K clicks from Bing:

So again, it was great to hear Satya’s statements about fair use and linking to publishers, but let’s face it, we need Google to do the same. That’s unless Bing starts gaining many users based on AI chat hitting the scene. I’ll cover that next.

By the way, it’s definitely worth noting that both Google and Bing are behind some other newer and smaller engines that are moving at a rapid pace with AI chat. For example, Neeva, You.com, and Perplexity.ai all provide AI chat, and have for some time. The great news is that they all  provide links to publishers from the chat window (letting users link downstream to find more information from the sources where the content originated from).

For example, here is NeevaAI providing an answer with my own content cited. It’s a weird feeling seeing this, but now after Bard’s lack of citations, it’s better than no citations at all. :)

Field, I mean SERP, of Dreams: If you build it, will they come? The user adoption issue.
As mentioned earlier, Bing killed it with their announcement and outdid Google. That was super-clear, but questions remain about how people will react. And I mean real people that use search engines for many tasks throughout the day. In other words, Bing AI chat looks pretty amazing, but will users follow?

When the buzz dies down, and you’re at home or work needing to search for something, will you quickly visit Google like you always do, or will you shift and visit Bing instead? On that note, Bing announced that they had one million people sign up for early access to the Bing AI chat experience in the first 48 hours after Tuesday’s presentation (see below). That’s great, but it’s not billions. And out of the one million, how many are just interested in seeing how it works, but will still rely on Google for most of their searches?

In addition, Bing’s mobile app skyrocketed up the charts based on people trying to gain early access to AI chat. It was one of the requirements for getting access to AI chat quicker than just being on the waitlist. But are people really using Bing’s mobile app or was it just a checkbox for gaining early access? So there was lots of buzz, but will that translate into more users? Nobody knows yet.

Bing also announced something interesting financially. They said for every 1% of share they gain, it equates to $2 billion in revenue. Here is a quote from Microsoft’s CFO for Windows and Search, Philip Ockenden, about the opportunity. Bret Kinsella covered this in his post about Google’s stock tanking after the Bard presentation on Wednesday, February 8th:

“For every one point of share gain in the search advertising market, it’s a $2 billion revenue opportunity for our advertising business.”

So, there’s a huge upside for Bing right now. Remember, Microsoft has many sources of revenue and Search is a relatively small one at this point. It’s profitable, but very small compared to Google. But that also gives them an advantage. They don’t have to worry as much about the search impact when testing big changes since they are driving massive revenue via other products and services. Google, on the other hand, earns a large percentage of revenue from Search. It’s much riskier to test big changes…

Back to usage, when asking my kids (18 and 16) if they would switch to Bing for AI chat, my daughter came back with a quick “maybe”. She said, “If it answers my questions quickly, I might give it a try…” My son on the other hand, said “hell no”. But after explaining more about Bing’s AI chat, he came around and said, “Well, I’d have to see it in action…” So there is hope for Bing. :)

Jarvis: Where the AI Search Juggernaut Is Ultimately Heading…
To wrap up this post, I wanted to quickly cover where I think AI Search is ultimately heading. And this shouldn’t be a surprise since we’ve seen it a thousand times in science fiction movies. I think it’s Jarvis from Iron Man.

If you’re not familiar with Jarvis, Tony Stark built an incredibly powerful personal AI assistant that he can call upon wherever he, is and for whatever he needs. Jarvis can answer any question and knows everything about Tony. It can answer any question thrown its way and can provide supporting information, or even take action on Tony’s behalf. See a quick video below:

So if you think about Jarvis for Search, then AI chat is the very beginning of that. Sure, some like Microsoft will vow to keep linking downstream to publishers, but isn’t that just a stopgap? Again, I have no idea how long it will take to get to the Jarvis stage of AI assistants, but in my opinion it’s coming. And at that point, I think we’ll all look back and laugh about having to enter keywords into a search engine, browsing endless search results pages, and pogosticking across sites trying to find an answer. But for now, I’d be much happier with AI chat providing links to my site (and to publishers across the web).

Summary: AI Chat in the search results has arrived, and publishers are watching closely.
AI chat has seen explosive growth since November of 2022 when ChatGPT launched, and we are all quickly going to learn how that impacts publishers. The great part is that the subsequent analysis will NOT be based on sample data and hypotheticals, but real searches by real people. And we’ll know fairly quickly if, and how, publishers can survive.

Since I help a lot of companies across verticals that rely on search traffic, I’ll be watching closely and analyzing the situation the best I can. Expect more posts from me soon on the impact of AI chat in Search based on real data. And believe me, I hope I can report that publishers are flourishing in this new AI world. Time will tell. Stay tuned.

GG

When The Hammer Falls – Analyzing Lyrics in the Google SERPs and Its Impact on Traffic [Case Study]

By 7 Comments

Summary: In the fall of 2014, both Bing and Google began surfacing song lyrics directly in the search engine results pages (SERPS). Since users could now find lyrics immediately in the SERPs, many wondered what would happen to lyrics websites that provided the same information, but required a click through to view the lyrics. This post provides findings from analyzing three large-scale lyrics web sites to determine the traffic impact of lyrics in the SERPs.

Song Lyrics Displayed In The Google Search Results

Article Contents and Quick Jumps:


Introduction
In April of 2014, I picked up a major algorithm update that heavily impacted lyrics web sites. The drop in traffic to many key players in the niche was substantial, with some losing 60%+ of their Google organic traffic overnight. For those of you familiar with Panda or Penguin hits, you know what this looks like.

Lyrics Web Sites Hit By Google Algorithm Update in April of 2014

I ended up digging in heavily and analyzing the drop across the entire niche. I reviewed a number of lyrics sites across several countries that got hit and wrote a post covering my findings (linked to above). After writing that post, I had a number of lyrics sites reach out to me for more information. They wanted to know more about what I surfaced, what the problems could be, and if I could help rectify the situation. It was a fascinating algo hit to analyze and I absolutely wanted to take on the challenge of helping the sites recover. So I began helping several of the lyrics sites that were heavily impacted.

2014 – A Crazy Year for Lyrics Sites
I took on several of the lyrics sites as clients and began heavily analyzing and auditing the negative impact. That included performing a deep crawl analysis of each site, a heavy-duty technical SEO analysis, a thorough content analysis, while also using every tool in my arsenal to surface SEO-related problems.

I won’t sugarcoat my findings, there were many problems I surfaced, across content, technical SEO, and even links (in certain situations). It was hard to say if the specific update in April was Panda, a separate algo update that hammered lyrics sites, or something else. But I tackled the situation by covering as many bases as I could. Each remediation plan was extensive and covered many ways to tackle the problems I surfaced. As time went on, and many changes were implemented, the sites started to recover. Some recovered sooner than others, while other sites took many more months to surge back.

Lyrics Website Recovering During Panda Update

On that note, many of the large lyrics sites have ridden the Panda roller coaster for a long time. And that’s common for large-scale websites that haven’t focused on Panda-proofing their web sites. Over time, insidious thin content builds on the site like a giant layer of bamboo. And as the bamboo thickens, Panda smells dinner. And before you know it, boom, Panda hits the site (and for these sites, it hit them hard).

After recovering, each site would hold their collective breath while subsequent Panda updates rolled out. Based on the lyrics web sites I have assisted, only one has fallen again to Panda. The others have remained out of the gray area and are doing well traffic-wise. Unfortunately, one lyrics web site I was helping saw a temporary recovery after recovering relatively quickly (almost too quickly). Quick recoveries are rare when you’re dealing with Panda, so I did find that specific recovery odd. It typically takes months before you see a major surge after being pummeled by Panda. The site surged during the 9/5 update and then got hammered again during the cloaked 10/24 update. And Panda has not rolled out since 10/24/14, so we’re still waiting to see if the site comes back.

Lyrics Website Temporary Recovery from Panda

But enough about Panda for now. Actually, Google Panda could pale in comparison to what showed up in late fall 2014. We all knew it was possible, considering Google’s ambition to provide more and more data in the search engine results pages (SERPs). But it’s another story when you actually see it happen. I’m referring to the search engines adding lyrics directly in the SERPs. You know, when someone searches for song lyrics, and boom, the lyrics show up right in the desktop or mobile SERPs. No click through needed. I’ll cover how this unfolded next.


Lyrics Show Up in the SERPs
Bing was the first to add lyrics in the SERPs on October 7, 2014. That was the first bomb dropped on lyrics sites. It was a small bomb, considering it was only showing in Bing in the United States and Bing has approximately 19.7% market share (according to comScore Dec 2014 stats). Bing also drives Yahoo search (organic and paid), but lyrics are not showing in Yahoo yet.

Lyrics in Bing SERPs

But the writing was on the wall. Lyrics were coming to Google, and sooner than later. When lyrics hit Bing, I sent emails to all of my lyrics clients explaining the situation, providing screenshots, and sample searches. Not every song would yield lyrics in the SERPs, but this was still a major event for the lyrics industry.

Next up was the first move by Google. On October 24, 2014, if you searched for a specific song, Google began providing a YouTube video with some song and artist information at the top of the SERPs. And near the bottom of that unit was a line or two from the lyrics and then a link to Google Play for the full lyrics. Whoa, so Google was beginning their assault on lyrics by simply linking to Google Play to view the lyrics. Again, I immediately emailed my clients and explained the situation, knowing lyrics were coming to the main SERPs soon.

Lyrics in Google SERPs Linking To Google Play

 

December 19, 2014 – The Hammer Falls
And then this happened:

Lyrics in Google SERPs Finally Arrive on December 19, 2014

And here was my Google+ share, which ended up getting a lot of attention:

Google Plus Share of Lyrics in the Google SERPs

 

I shared this screenshot of Google including lyrics directly in the SERPs, and the G+ post got noticed, a lot. That share was mentioned on a number of prominent websites, including Search Engine Roundtable, TechCrunch, Billboard, and more.

To clarify what was happening search-wise, on December 19, 2014 Google began showing song lyrics for users in the United States, and only for certain songs. I’m assuming the limit on songs and geography was based on licensing, so this doesn’t impact every song available. I’ll cover more about the impact of those limitations soon when I dig into some stats, but it’s an important note.

For example, if you search for “bang bang lyrics” in the United States, you get this:

Bang Bang Lyrics in US Google SERPs

But if you search for “you shook me all night long lyrics”, you won’t see lyrics in the SERPs. Clearly Google doesn’t have the rights to present the lyrics to all AC/DC songs, but it does for “Bang Bang”.

You Shook Me All Night Long Without Lyrics in US Google SERPs

And by the way, that’s for the desktop search results. This is also happening in mobile search, in the United States, and for certain songs. Talk about dominating the mobile SERPs, check out the screenshot below. Where on desktop, you get the lyrics, but still see links to lyrics websites above the fold (typically), mobile is another story.

Check out the search for “bang bang lyrics” on my smartphone:

Bang Bang Lyrics in the Mobile U.S. Google SERPs

Can you see the massive difference? It’s just lyrics, and nothing else. And to add insult to injury, the percentage of users searching for lyrics is heavily skewed mobile. And that makes sense. Those users are on the go, hear a song, want to know the lyrics, and simply search on their phones. Or, they are in a situation where their phone –is their computer– so their searches will always be mobile.

Mobile Heavy Queries for Lyrics Globally

 

Death to Lyrics Websites?
Based on what I’ve explained so far, you know that Panda loves taking a bite out of lyrics web sites and you also know that both Google and Bing are providing lyrics directly in the SERPs (in the US and for certain songs). And you might guess that all of this means absolute death for lyrics websites. But wait, does it? I wouldn’t jump to conclusions just yet. There are definitely nuances to this situation that require further analysis and exploration.

For example, how much of a hit have the lyrics sites taken based on lyrics in the SERPs? How much traffic dropped for each song that yields lyrics in the SERPs? Was there an impact just in the United States or around the world too? And what about the difference between desktop and mobile? All of these were great questions, and I was eager to find answers.

So, I reached out to several of my lyrics clients and asked if I could analyze the changes and document the data in this post (anonymously of course). The post isn’t meant to focus on the sites in particular, but instead, focus on the impact that “lyrics in the SERPs” have made to their traffic. The lyrics websites I’ve been helping generate revenue via advertising, so a massive drop in traffic means a massive drop in revenue. It’s pretty much that simple at this point. That’s why Panda strikes fear in every lyrics web site owner and why lyrics in the SERPs can strip away visits, pageviews, and ad dollars. It’s a new one-two punch from Google.


Analyzing Three Large-Scale Lyrics Websites
Three of my clients were nice enough to let me move forward with the analysis. And I greatly appreciate having clients that are awesome, and are willing to let me analyze and share that data. The three sites I analyzed for this post are large-scale lyrics sites. Combined, they drive more than 30 million visits from Google organic per month and have approximately 6 million lyrics pages indexed. And as I explained earlier, a lot of that traffic is from users on mobile devices. Approximately 40-50% of all Google organic traffic is from mobile devices (across all three sites).

Process:
My goal with the analysis was to understand the impact of lyrics in the SERPs from a click-through and traffic standpoint. I dug into search queries driving traffic over time to all three sites while also checking impressions and clicks in the SERPs (via Google Webmaster Tools, both desktop and mobile). Then I also checked Google Analytics to determine the change in traffic levels to song pages since the lyrics hit the SERPs.

For example, if a query saw a similar number of impressions since the launch of lyrics in the SERPs, but clicks dropped off a cliff, then I could dig in to analyze the SERPs for that query (both desktop and mobile). I found some interesting examples for sure, which I’ll cover below.

An example of stable or increasing impressions, but clicks dropping off a cliff: 

Google Webmaster Tools Impressions and Clicks for Lyrics Queries

 

Caveats:
My analysis measured the impact right after lyrics hit the SERPs (from December 19, 2014 through the end of January 2015). The holidays were mixed in, which I tried to account for the best I could. Some of the lyrics sites saw steady traffic during the holidays, while one dipped and then returned as the New Year approached. The songs I analyzed and documented were not holiday-focused songs. I made sure to try and isolate songs that would not be impacted by the holidays. Also, Google Webmaster Tools data was sometimes wonky. I’m sure that’s no surprise to many of you working heavily in SEO, but it’s worth noting. I tried my best to exclude songs where the data looked strange.

Google Webmaster Tools & Advanced Segmentation in GA
When I began my analysis, I quickly found out that the straight reporting in both Google Webmaster Tools and Google Analytics wouldn’t suffice. Overall Google organic traffic wouldn’t help, since lyrics only rolled out in the SERPs in the United States. When checking traffic since the rollout, you really couldn’t see much overall change. But the devil is in the details as they say. So I used the functionality available to me in both GWT and GA to slice and dice the data. And that greatly helped me understand the impact of lyrics in the SERPs.

In Google Webmaster Tools, the search queries reporting enables you to filter the results. This was incredibly helpful, as I was able to isolate traffic from the United States and also view web versus mobile traffic. But there was another nifty filter I used that really helped. You see, many people visit lyrics websites for the meaning of the lyrics, and not just to see the lyrics. For example, “take me to church meaning” or “meaning of hallelujah lyrics”.

The reason I wanted to weed those queries out is because as of now, Google does not provide the lyrics in the SERPs for “meaning” focused queries. And that’s good for my clients by the way. So by adding the filters per site, I would able to isolate songs that could be impacted.

Filtering GWT Search Queries by Search Property, Location, and Negative Query:

Google Webmaster Tools Filters for Property, Location, and Query

After setting the filters, I was able to search for queries that yielded relatively stable impressions, but saw a drop in clicks and click through rate. And I always kept an eye on average position to make sure it didn’t drop heavily.

From a Google Analytics standpoint, I ran into a similar problem. Top-level statistics wouldn’t cut it. I needed Google organic traffic from the United States only. And then I wanted both Desktop and Mobile Google organic traffic from the United States only (separated). That’s where the power of advanced segments come in.

I built segments for Desktop Google organic traffic from the United States and Mobile Google organic traffic from the United States. By activating these segments, my reporting isolated that traffic and enabled me identify trends and changes based on those segments alone. By the way, I wrote a tutorial for how to use segments to analyze Panda hits. You should check that out if you aren’t familiar with segments in GA. You’ll love them, believe me.

Filtering Google Organic Traffic from the United States in GA Using Segments:

Google Analytics Segments for U.S. Desktop Google Organic Traffic

 

So, with the right tools and filters in place, I began to dig in. It was fascinating to analyze the queries leading to all three sites now that lyrics hit the SERPs. I cover what I found next. By the way, this posts focuses on Google and not Bing. I might write up another post focused on Bing’s lyrics in the SERPs, but I wanted to focus on Google to start.


The Impact of Lyrics in the SERPs – The Data
With multiple computers up and running, two phones, and two tablets, I began to dig in. I wanted to find queries and songs that typically drove traffic to the three sites that now yielded lyrics in the SERPs. And then I wanted to see what happened once those lyrics hit the SERPs, the impact on clicks, traffic, etc. I have documented a number of examples below. By the way, there are many more examples, but I wanted to just provide a sampling below. Here we go…

 

Spill The Wine Lyrics by War
Google Organic Desktop US Traffic Down 73%
Google Organic Mobile US Traffic Down 65%
GWT Clicks Down 56%

 

Sister Ray Lyrics by The Velvet Underground
Google Organic Desktop US Traffic Down 73%
Google Organic Mobile US Traffic Down 56%
GWT Clicks Down 84%

 

Rude Lyrics by Magic!
Google Organic Desktop US Traffic Down 41%
Google Organic Mobile US Traffic Down 32%
GWT Clicks Down 55%

 

Bang Bang Lyrics by Jesse J, Nicki Manaj and Ariana Grande
Google Organic Desktop US Traffic Down 32%
Google Organic Mobile US Traffic Down 47%
GWT Clicks Down 66%

 

Fireproof Lyrics by One Direction
Google Organic Desktop US Traffic Down 44%
Google Organic Mobile US Traffic Down 40%
GWT Clicks Down 29%

 

All of Me Lyrics by John Legend
Google Organic Desktop US Traffic Down 39%
Google Organic Mobile US Traffic Down 14%
GWT Clicks Down 61%

 

Country Road Lyrics by John Denver
Google Organic Desktop US Traffic Down 62%
Google Organic Mobile US Traffic Down 45%
GWT Clicks Down 36%

 

Come Sail Away Lyrics by Styx
Google Organic Desktop US Traffic Down 43%
Google Organic Mobile US Traffic Down 27%
GWT Clicks Down 55%

 

Midnight Special Lyrics by Huddie William Ledbetter
Google Organic Desktop US Traffic Down 53%
Google Organic Mobile US Traffic Down 85%
GWT Clicks Down 33%

 

Comfortably Numb Lyrics by Pink Floyd
Google Organic Desktop US Traffic Down 46%
Google Organic Mobile US Traffic Down 17%
GWT Clicks Down 43%

 

Yes, There’s A Serious Impact
As you can see from the statistics above, both desktop and mobile traffic to the song pages dropped significantly since lyrics hit the SERPs (for songs that yield lyrics in the SERPs). Again, these songs showed stable impressions during the timeframe, yet showed large drops in clicks from the SERPs, and subsequent traffic to the three lyrics sites I analyzed.

Some users were clearly getting what they wanted when searching for lyrics and finding that information in the SERPs. And in mobile search, the lyrics take up the entire results page. So it’s no surprise to see some mobile numbers absolutely plummet after lyrics hit the SERPs.


What Could Lyrics Sites Do?
Above, I provided a sampling of what I saw while analyzing the impact of lyrics in the U.S. Google SERPS. Clearly there’s a large impact. The good news for lyrics sites is that there are several core factors helping them right now.

  • This is only in the United States.
  • The lyrics only trigger when the query is structured in certain ways. For example, “magic rude lyrics” yields lyrics where “rude lyrics magic” does not. Also, if additional words are entered in the query, lyrics will not be shown (like “meaning” which I explained earlier.)
  • Not all songs are impacted (yet). I found many examples of songs that did not yield lyrics in the SERPs. Again, this is probably due to licensing issues.

If you look at the overall traffic numbers for the sites I analyzed (and the other sites I have access to), Google organic traffic overall has not been heavily impacted. Taking all global Google organic traffic into account, and across all songs, you clearly don’t see the huge drop like I showed you for the songs listed above. That said, this is still a grave situation for many lyrics sites. The content they have licensed and provided on their sites is now being surfaced directly in the SERPs. If this expands to more songs, more countries, and for additional queries, then it can have a massive impact on their businesses. Actually, it could very well end their businesses.

Moving forward, lyrics sites need to up their game from a functionality and value proposition standpoint. If Google can easily add lyrics to the SERPs, then lyrics sites need to keep driving forward with what Google can’t do (at least for now). They should develop new functionality, strengthen community engagement, provide member benefits, include more data and media for artists and songs, provide a killer mobile experience, etc.

Remember, there are many people searching for additional information related to songs. For example, people want to know the meaning of lyrics and seem to enjoy the community engagement about learning what each lyric means. And lyrics don’t trigger in the SERPs for those queries (yet).

And then you have the next generation of devices, social networks, messaging apps, gaming consoles, connected cars, etc. I would start thinking about how people are going to search for lyrics across new devices and in new environments. That’s a new frontier and it would be smart to begin building and testing lyrics applications that can work in those new environments. Mobile, wearables, voice search, cars, etc. provide a wealth of opportunity for business owners focused on music. It just takes the right ideas, time, resources, and of course, money.

But I’ll stop there. I think that topic can be an entire post and this one is getting too long already. :)

 

Summary – Moving Forward With (Expanding) Lyrics in the SERPs
In the short-term, it’s hard to say how this will expand. Google and Bing might drop the effort and keep things as-is, or they could keep expanding lyrics in the SERPs until every song and every country is covered.

Based on the current song and geography limits in Google and Bing, lyrics websites are still surviving, and especially for searches outside the United States. It will be interesting to watch this space over time, especially since I have several clients adapting to the new lyrics world as I write this post.

From an SEO standpoint, between Google Panda and content surfacing in the SERPs, lyrics web sites are fighting a battle on two fronts. If it’s not Panda attacking the site one night, it’s the Knowledge Graph pushing song lyrics front and center in the SERPs. And in this day and age, wars are won by technology, not brute strength. So lyrics sites need to up their engineering prowess, think two to three steps ahead of the industry, and then execute quickly and at a very high level.

That’s how they can survive and prosper in the coming years. Of course, that’s until we have a Google chip implanted in our brains that instantly provides the lyrics to every song ever written, from the around the world, since the beginning of time. Think about that for a second.

GG

 

Panda, Penguin, and Manual Actions – Questions, Tips, and Recommendations From My SES Atlanta Session

By 12 Comments

SES Atlanta Panda

{Important Update About Penguin: Read John Mueller’s latest comments about the Penguin algorithm.}

I just returned from SES Atlanta, where I presented “How To Avoid and Recover From Panda, Penguin, and Manual Actions”. The conference was outstanding, included a killer keynote by Duane Forrester and sessions packed with valuable information about SEO and SEM. By the way, I entered my hotel room in Atlanta and immediately saw a magazine on the desk. The photo above is the cover of that magazine! Yes, a Panda was on the cover. You can’t make this stuff up. :)

During (and after) my presentation about algorithm updates and penalties, I received a number of outstanding questions from audience members. And later in the day, I led a roundtable that focused on Panda and Penguin. There were also some great conversations during the roundtable from business owners and marketers across industries. It’s always interesting to hear top marketer concerns about major algorithm updates like Panda and Penguin (and especially Panda 4.0 which just rolled out in late May). We had a lively conversation for sure.

On the plane flight home, I started thinking about the various questions I was asked, which areas were the most confusing for marketers, and the tips and recommendations I was sharing.  And based on that list, I couldn’t help but think a Q&A style blog post could be very helpful for others dealing with Panda, Penguin, and manual actions. So, I decided to write this post covering a number of those questions. I can’t cover everything that I spoke about at SES Atlanta (or this post would be huge), but I can definitely provide some important tips and recommendations based on questions I received during the conference.  Let’s jump in.

Algorithm Updates and Manual Actions – Q&A From SES Atlanta

Question: I’ve been hit by Panda 4.0. What should I do with “thin content” or “low-quality” content I find on my website?  Is it better to nuke the content (404 or 410), noindex it, or should I redirect that content to other pages on my site?

Glenn: I hear this question often from Panda victims, and I know it’s a confusing topic. My recommendation is to remove thin and low-quality content you find on your site. That means 404 or 410 the content or noindex the content via the meta robots tag. When you have a content quality problem on your site, you need to remove that content from Google’s index. In my experience with helping companies recover from Panda, this has been the best path to take.

That said, if you find content that’s thin, but you feel you can enhance that content, go for it. If you believe the content could ultimately hold information that people are searching for, then beef it up. Just make sure you do a thorough job of developing the additional content. Don’t replace thin content with slightly thin content. Create killer content instead. If you can’t, then reference my first point about nuking the content.

Also, it’s important to ensure you are removing the right content… I’ve seen companies nuke content that was actually fine thinking it was low-quality for some reason. That’s why it’s often helpful to have an objective third party analyze the situation. Business owners and marketers are often too close to their own websites and content to objectively rate it.

Panda Decision Matrix

 

Question: How come I haven’t seen a Panda recovery yet even though I quickly made changes? I was expecting to recover during the next Panda update once the changes were implemented.

Glenn: This is another common question from Panda victims. It’s important to understand that completing the changes alone isn’t enough. Google first needs to recrawl the site and the changes you implemented.  Then it needs to better understand user engagement based on the changes. I’ve explained many times in my blog posts about Panda that the algorithm is heavily focused on user engagement. So just making changes on your site doesn’t provide Google enough information.

Panda recovery can take time. Just read my case study about 6 months with Panda. That was an extreme situation in my opinion, but it’s a great example of how long it can take to recover.

Second, Panda roughly rolls out once per month. You need an update to occur before you can see changes. But that’s not a hard rule. John Mueller from Google clarified the “Panda Tremors” I have been seeing since Panda 4.0, and explained that there isn’t a fixed frequency for algorithm updates like Panda. Instead, Google can continue to tweak the algo to ensure it yields the desired results. Translation: you might see turbulence after a Panda hit (and you may see increases or decreases as the tremors continue).

Panda Tremors John Mueller

And third, you might see smaller recoveries over time during subsequent updates (versus a full recovery in one shot). I’ve had several clients increase with subsequent Panda updates, but it took 4-5 updates for them to fully recover. So keep in mind that you might not see full recovery in one shot.

 

Question:  We know we have an unnatural links problem, and that we were hit by Penguin, but should we tackle the links problem or just build new links to balance out our link profile?

Glenn: I’ve seen many companies that were hit by Penguin avoid tackling the root problem, and instead, just try and build new links to balance out their link profile. In my opinion, that’s the wrong way to go. I always recommend aggressively handling the unnatural links situation, since that’s the most direct path to Penguin recovery.

And to clarify, you should still be pumping out killer content, using Social to get the word out, etc. I always tell clients impacted by Penguin or Panda to act like they aren’t impacted at all. Keep driving forward with new content, sharing via social media, connecting with users, etc. Fresh links and shares will be a natural side effect, and can help the situation for sure. And then the content they are building while under the Penguin filter could end up ranking well down the line. It’s hard to act like you’re not hit, but that’s exactly what you need to do. You need to be mentally tough.

Address Unnatural Links for Penguin

 

Question: Is it ok to remove content from Google’s index? Will that send strange signals to the engines?

Glenn: Nuke it. It’s totally fine to do so, and I’ll go even further and say it could be a great thing to do. I mentioned this several times in my Panda 4.0 findings, but the right indexation is more important than high indexation. In other words, make sure Google has your best content indexed, and not thin, duplicate, or other low-quality content.

I had one client drop their indexation by 83% after being impacted by Phantom and Panda, and they are doing extremely well now Google organic-wise. I love the screenshot below. It goes against what many marketers would think. Lower indexation = more Google traffic. That’s awesome.

Indexation and Panda

 

Question: We consume a lot of syndicated content. What’s the best way to handle attribution?

Glenn: I saw a number of sites get smoked during Panda 4.0 that were consuming a lot of syndicated content and not handling that properly SEO-wise. The best way to handle attribution for syndicated content is to use the cross domain canonical url tag pointing to the original article. If you can’t do that (or don’t want to do that), then you can keep the content out of Google’s index by noindexing it via the meta robots tag.

It’s not your content, so you shouldn’t be taking credit for it.  That said, if set up correctly, it’s fine to have syndicated content on your site for users to read. But the proper attribution is important or it can look like you are copying or scraping content. I know that won’t go over well for ad teams looking to rank in organic search (to gain more pageviews), but again, it’s not your content to begin with.

Syndication and Panda

 

Question: Why hasn’t there been a Penguin update since October of 2013? What’s going on? And will there ever be another update?

Glenn: It’s been a long time since the last Penguin update (October 4, 2013). Like many others heavily involved with Penguin work, I’m surprised it has taken so long for another update.

Penguin 2.1 on October 4, 2013

Matt Cutts recently explained at SMX Advanced that they have been heavily working on Panda 4.0, so Penguin has taken a back seat. But he also said that an engineer came up to him recently and said, “it’s probably time for a Penguin update”. That situation is both positive and scary at the same time.

On the one hand, at least someone is thinking about Penguin on the webspam team! But on the flip side, they clearly haven’t been focusing on Penguin for some time (while many Penguin victims sit waiting for an update). On that note, there are many webmasters who have rectified their unnatural link problems, disavowed domains, urls, etc., and are eagerly awaiting a Penguin update. It’s not exactly fair that Google has been making those business owners wait so long for Penguin to roll out again.

Now, there’s always a possibility that there is a problem with the Penguin algorithm. Let’s face it, there’s no reason it should take so long in between updates. I’m wondering if they are testing Penguin and simply not happy with the results. If that’s the case, then I could see why they would hold off on unleashing a new update (since it could wreak havoc on the web). But that’s just speculation.

In my opinion, it’s not cool to let Penguin victims that have worked hard to fix their link problems sit in Penguin limbo. So either Google is seriously punishing them for the long-term, they have put the algo on the back burner while focusing on other algos like Panda, or Penguin is not up to par right now. Remember, if Google isn’t happy with the results, then they don’t have to push it out. And if that’s the case, Penguin victims could sit in limbo for a long time (even longer than the 9 months they have waited so far.)  Not good, to say the least.


Important Penguin Update: Google’s John Mueller provided more information about the Penguin algorithm on today’s Webmaster Central Office Hours Hangout.

John was asked if Penguin would be released again or if it was being retired. And if it was being “retired”, then would Google at least run it one more time to free those webmasters that had cleaned up their link profiles. John explained that Penguin was not being retired. Let me say that again. he said Penguin is not being retired. John explained that it can sometimes take longer than expected to prepare the algorithm and update the necessary data. He also explained that if Google were to retire an algorithm, then they would “remove it completely” (essentially removing any effect from the algorithm that was in place).

So we have good news on several fronts. Penguin is still alive and well. And if Google did retire the algo, then the effect from Penguin would be removed. Let’s hope another Penguin update rolls out soon.

You can view the video below (starting at 5:16) or you can watch on YouTube -> https://www.youtube.com/watch?v=8r3IIPCHt0E&t=5m16s

 

Question: We’ve been hit by both Panda and Penguin. We don’t have a lot of resources to help with recovery, so which one do we tackle first?

Glenn: I’ve helped a number of companies with Pandeguin problems over the years, and it’s definitely a frustrating situation for business owners. When companies don’t have resources to tackle both situations at the same time, then I’ve always been a big fan of tackling the most acute situation first, which is Penguin.

Pandeguin Hit

Panda is a beast, and has many tentacles. And Penguin is all about unnatural links (based on my analysis of over 400 sites hit by Penguin since April 24, 2012). That’s why I recommend focusing on Penguin first (if you can’t knock out both situations at once). I recommend aggressively tackling unnatural links, remove as many spammy links as you can, and then disavow the remaining ones you can’t get to manually. Then set up a process for monitoring your link profile over time (to ensure new unnatural links don’t pop up).

After which, you can tackle the Panda problem. I would begin with a comprehensive Panda audit, identify the potential problems causing the Panda hit, and aggressively attack the situation (the bamboo). Move quickly and aggressively. Get out of the grey area of Panda (it’s a maddening place to live).

 

Question: Is linkbuilding dead? Should I even focus on building links anymore and how do I go about doing that naturally?

Glenn: Links are not dead! The right links are even more important now. I know there’s a lot of fear and confusion about linkbuilding since Google has waged war on unnatural links, but to me, that makes high quality links even more powerful.

Duane Forrester wrote a post recently on the Bing Webmaster Blog where he explained if you know where a link is coming from prior to gaining that link, then you are already going down the wrong path. That was a bold statement, but I tend to agree with him.

Duane Forrester Quote About Linkbuilding

I had several conversations about this topic at SES Atlanta. To me, if you build killer content that helps your target audience, that addresses pain points, and teaches users how to accomplish something, then there’s a good chance you’ll build links. It’s not the quantity of links either… it’s the quality. I’d rather see a client build one solid link from a site in their niche versus 1000 junky links. The junky links are Penguin food, while the solid link is gold.

 

Question: I was hit by Panda, but my core competitors have the same problems we do. We followed what they were implementing, and we got hit. Why didn’t they get hit? And moving forward, should we follow others that are doing well SEO-wise?

Glenn: I can’t tell you how many times companies contact me and start showing me competitors that are doing risky things SEO-wise, yet those sites are doing well in Google. They explain that they tried to reproduce what those competitors were doing, and then they ended up getting hit by Panda. That situation reinforces what I’ve told clients for a long time. Competitive analyses can be extremely beneficial for gathering the right intelligence about your competitors, but don’t blindly follow what they are doing. That’s a dangerous road to travel.

Instead, companies should map out a strong SEO strategy based on their own research, expertise, target audience, etc. Ensure you are doing the right things SEO-wise for long-term success. Following other companies blindly is a dangerous thing to do. They could very easily be headed towards SEO disaster and you’ll be following right along.

For example, I had a client always bring up one specific company to me that was pushing the limits SEO-wise (using dark grey hat tactics). Well, they finally got hit during a Panda update in early 2014 and lost a substantial amount of traffic. I sent screenshots to my client which reinforced my philosophy. My client was lucky they didn’t follow that company’s tactics… They would have jumped right off the SEO cliff with them. The screenshot below shows an example of a typical surge in Google before a crash.

Surge in Traffic Before Algo Hit

 

Question: We’ve been working hard on a manual action for unnatural links, but right before filing reconsideration, it expired. What should we do?

Glenn: I’ve seen this happen with several clients I was helping with manual actions. It’s a weird situation for sure. You are working on fixing problems based on receiving a manual action, and right before you file a reconsideration request, the manual action disappears from Google Webmaster Tools. When that happens, is the site ok, do you still need to file a reconsideration request with Google, should you wait, or should you continue working on the manual action?

It’s important to know that manual actions do expire. You can read that article by Marie Haynes for more information about expiring manual actions. Google has confirmed this to be the case (although the length of each manual action is variable). But those manual actions can return if you haven’t tackled the problem thoroughly… So don’t’ think you’re in the clear so fast.

Expiring Manual Actions

 

That said, if you have tackled the problem thoroughly, then you are probably ok. For example, I was helping a company with a manual action for unnatural links and we had completed the process of removing and disavowing almost all of their unnatural links. We had already written the reconsideration request and were simply waiting on a few webmasters that were supposed to take down more links before filing with Google.

As we were waiting (just a few extra days), the manual action disappeared from Google Webmaster Tools. Since we did a full link cleanup, we simply drove forward with other initiatives. That was months ago and the site is doing great SEO-wise (surging over the past few months).

Just make sure you thoroughly tackle the problem at hand. You don’t want a special surprise in your manual action viewer one day… which would be the return of the penalty. Avoid that situation by thoroughly fixing the problems causing the penalty.

 

Summary – Clarifying Panda and Penguin Confusion
As you can see, there were some outstanding and complex questions asked at SES Atlanta. It confirms what I see every day… that business owners and webmasters are extremely confused with algorithm updates like Panda and Penguin and how to tackle penalties. And when you combine algo updates with manual actions, you have the perfect storm of SEO confusion.

I hope the Q&A above helped answer some questions you might have about Panda, Penguin, and manual actions. And again, there were several more questions asked that I can’t fit into this post! Maybe I’ll tackle those questions in another post. So stay tuned, subscribe to my feed, and keep an eye on my Search Engine Watch column.

And be prepared, I felt a slight chill in the air this past weekend. The next Penguin update could (and should) be arriving soon. Only Google knows, but I hope they unleash the algo update soon. Like I said in my post, there are many webmasters eagerly awaiting another Penguin rollout. Let’s hope it’s sooner than later.

GG