I can’t even pinpoint within my data as to when this foolish or unfortunate incident occured because well… just look at the data.

That is nigh on 5 years of Google Blogsearch referral data, though because of the long time range it is listed as sampled data. There may be some traffic sources I have missed – variations of url depending on how Google were displaying blogsearch pages, but that is referrals from http://blogsearch.google.com

• In the past when I have checked at various times (though admittedly it has been a while) I had ranked well in blogsearch.
• There have always been so few referrals that I have more or less ignored Blogsearch
• The only blogsearch I have used are for links to me – in the WordPress interface, and occasionally to grab more results than WordPress displays – it has always been unreliable
• Blogsearch picks up links from all kinds of things
  • blogroll links
  • comments
  • if you send a pingback and a blog displays them, it will come up in blogsearch too

Because of all of these factors I had always assumed that the idea of using noindex on a feed of any kind was to prevent that feed appearing in Google’s primary organic results.

A pretty Feedburner feed isn’t a terrible landing page, but it is possble to do better. I have even written about using my feedburner URL when leaving blog comments in the past, as in some ways it immediately signals you want people to subscribe more so than linking to a blog.

Other RSS search engines were indexing my feed content – Technorati, Blogcatalog, Icerocket – my feeds were being read by my readers, picked up by various Twitter robots etc.

And of course my content remained indexed in Google’s primary organic index.

But then a few days ago I was browsing a little and looking for additional sources for a story followon, and noticed I wasn’t listed for previous coverage. I hadn’t been specific in the title that I was related… but there wasn’t a lot of competition.

Then I discovered this:-

My first thought for 5 minutes was somehow for some crazy reason I had become penalized in Google Blogsearch – then I rationalized it in thinking it must be something to do with noindex settings in Feedburner.

You see I had never equated noindex with a blog search engine – every other blog search engine which sent me traffic was still picking up my content and sending me traffic.

Google Indexing RSS Feeds

There is still a very real need for a way to tell Google…

“Hey Google, this is my RSS feed – you can index it for Google Blog Search, but I don’t want it to appear in the organic search results.”

3 years ago Google were saying they were working to remove RSS feeds from organic search.
3 years later feeds from Feedburner are still appearing in organic search results.

http://feeds.feedburner.com all the results seem to have been removed
http://feeds2.feedburner.com there still seems to be plenty of feeds within the search results

Information About & Help With Feedburner Since Google Acquisition

On a scale of 1 to 10 Google Feedburner Support gets a 2 – it is a free service, Google monetize it providing Adsense for feeds, but don’t expect anyone to answer support queries in the Google groups from Feedburner.
Documentation is sparse – hardly updated in the over 3 years since Google bought Feedburner… but then there haven’t been too many visible changes other than adding Adsense. I am sure there have been changes to help with scaling, especially how it eventually was made easier to integrate with Blogspot, but very little for anyone else.

Feedburner Noindex Controls

So this I believe is the culprit

This is the code that gets added to the RSS feed.

<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" /><meta xmlns="http://pipes.yahoo.com" name="pipes" content="noprocess" />

That data is still not transferred to feed items that are shared within Google Reader or feeds such as tags created there – which can get fed to other places and indexed.

The left hand doesn’t know what the right hand is doing

I have explained my whoopsie, but somewhere in the Googleplex they are a little confused over what they are doing as well.

Blogsearch isn’t the only search for my Blog posts

For instance there is Google Buzz

Now remember – Google is treating the noindex on my RSS feed as being an instruction to not include my content in Google BlogSearch…. so you would expect that instruction to be universal for the RSS content.

Those were taken from the PUBLIC timeline of Buzz. That is content that Google isn’t indexing on Blogsearch due to a noindex in the XML.

I also have my full content being fed into Facebook and being indexed and made searchable within Facebook, but at least that is my choice.

The only way to prevent content being shared and indexed is currently to block Google Reader from accessing feeds. I have been trying for over 4 years to get Google to introduce more publisher controls for sharing… as it would be easy to share private content from Google Reader by mistake… with Pubsubhubbub it can be broadcast by mistake to your 1000s of Buzz subscribers instantly.
This is possibly why Google have never introduced support for http authentication.

With their current stance for sharing freedoms, it doesn’t make sense for them to treat the current xml declarations as an instruction not to index the content in Blogsearch, as the content is in Buzz anyway. It should be treated as just a noindex for the page.
Alternatively they should add support for x-header noindex, then noindex in the XML would be for search engines, and it should travel with each content item, even to Buzz, possibly with no way to share the content.

Vanessa Fox came across something interesting in her vanity search today, and I thought I would add to the mystery and mystique of… the Google SERPs.

I noticed that within my own not quite so vanity searches ;) call it an ego search, that a recent post by Tinu about the paid links problems was ranking highly (#7).

That isn’t overly strange, until you look a little deeper in the SERP, and discover a second listing for exactly the same article, at exactly the same URL, at #27

As far as I am concerned this in many ways confirms Vanessa’s observations, with a recent blog search result being pulled into the main SERPs, as Tinu’s post ranks highly in blogsearch (not the newest, but one is on my domain, and another is a comment on Robert Scoble’s blog)

Thus Google might well now be including one recent blog post in universal search for some search queries.

But… Yeah There Always Is One… or Two, or Three

• This Only Appears In Personal Searches

  I logged out of Google to double check, and unfortunately the listing was cleaner without personal results.

• This Doesn’t Seem To Apply To Blue Pills

  I tried a few more competitive search terms, and could find no correlation between blog search and normal search, although this might be hampered a little because those are not terms Google might think should return a personalized result for, because they have no history.

• This Doesn’t Apply To Other Competitive Terms

  I tested quite a few which I have searched in the past, and of the competitive terms I found no evidence to suggest that blogs were being used as a source.

Vanessa’s results from her post also appear in my own SERPs, even the personal ones, so what I am seeing with my “ego” searches might not be the same as Vanessa’s “vanity” searches.

WordPress SEO Tips & Hacks

Subscribe to Comments Hacks – I am including this at the top of the list because it is highly important. Emails sent out from your domain could cause you problems with email delivery or spam complaints, and the plugin by default doesn’t comply with CAN-SPAM if you or someone else (a spammer) includes a marketing message in your emails.
The danger of overloading your server with emails being sent faster than your host allows is also worth thinking about. I know multiple blogs that have problems delivering all subscribed comments and I am sure the owners are not aware of this.

Just imagine you have 20 people comment on a post, and then you get 5 new comments in the space of one hour – that would put you over the limit on most shared hosting.

Mine are set up to be delivered by Gmail by SMTP – which email host is going to block Gmail totally?

Spam Karma latest release, and I also speculated that I get less spam using it than if I was using Akismet.

My Ultimate List of Dofollow Plugins first published in February still gains links on a daily basis, and has helped convert 100s of people into removing nofollow from their comments and trackbacks.
I have a followup project planned that I will launch within the next week that will provide a way of highlighting blogs that wish to be highlighted as supporting DoFollow.
The page also now includes information on how to remove NoFollow from blogs using Blogger and I hope to be able to include detailed Typepad and MoveableType instructions (I know of 3 people working on this)

How linking to social media and bookmarking sites could affect your search engine performance. The discussion that followed on is just as informative as the post.

Tagging and How Tagging Can Help SEO is one of the primary topics of this blog. I have written a lot of posts regarding the benefits of using Ultimate Tag Warrior (that is a link to the tag).
My analysis of the Google BlogSearch Patent didn’t gain a lot of links from SEO blogs, but then most SEO bloggers don’t understand the benefits of tagging and are too worried about duplicate content (another tag link)

Do you get linked to by the press? I also highlighted how bad some press agencies are with linking.

My attack on Top Commenters plugins caused some uproar. It is great to provide a way to thank the people who comment on your blog, but I prefer to do that by using DoFollow.
If you can understand my milk bottle analogy, you will go a long way to improving the SEO profile of your blog. If you do use a Top Commenters plugin, at least include it using some WordPress logic functions such as is_home() and is_single()

Top WordPress Plugins

I will be opening up the voting for the first round of my cash donations for best wordpress plugins later today (I need to decide which plugin or service to use). I am giving away real money in large chunks to help support WordPress Plugin Development.

Monetization & Blogging Tips

Warning about using Amazon Contextual Links

PayPerPost vs Jason Calacanis turned into an in depth discussion regarding ThisNext vs PayPerPost on Disclosure (you really have to read the comments)

Performancing Metrics Is a Wolf in Sheep’s Clothing for Niche Website owners and why I recommend Clicky instead.

I write a lot about disclosure in affiliate marketing and blogging. Whilst PayPerPost is constantly being highlighted, the people doing the highlighting are actually as guilty of poor disclosure and unethical practices themselves.

Maki asked a number of bloggers why they display a feed counter and I gave what I believe is the most in depth answer. There are some good tips there for those people who already have an existing mailing list to jumpstart their blog subscription for an insane amount of social proof.
Why don’t Aweber and Getresponse offer a decent email subscription to RSS feeds that reports numbers to Feedburner? Anyone who uses their services misses out on the social proof benefit.
They should also offer their own social proof icons for email subscriptions.

In some ways I hate product launch formula style marketing of services. The scarcity factor makes it hard for a potential buyer to make an informed decision despite all the proof.
Sometimes the materials left over are of huge value, and I recommend you watch the videos from Rich Schefren.

Product Reviews

Stampede Traffic

Sponsored Reviews – with this service you can negotiate with me for paid reviews :)

Volusion Shopping Cart – was I too harsh on them?
Auction Ads

Web 2.0

Is this the Future of MyBlogLog when we finally get an API? Smart internet marketers should already be planning on how the data could be best integrated into other applications.

Tips on using Reddit and Stumbleupon. The StumbleUpon post lists my membership details to a number of social networks… lets be friends. I really should include those in my sidebar.

Sometimes it is amazing the knock-on effect of what you blog about.

My How To Use MyBlogLog post sent Cord some traffic, but the important factor is the quality of traffic sent. Eric from MyBlogLog saw the post, read Cord’s 2 part guide, and then listed it on the MyBlogLog blog, along with some more tips.

I wrote a fair amount about Twitter, partially because it was the talk of the blogosphere, and partially because as with all emerging technology, there were a lot of ethical problems in using it for commercial use. The best way to read full coverage is to use my Twitter tag.

I love Technorati, but they really have problems coping with all the new kinds of RSS and micro blog services such as Twitter, and co.Comment

I had some Digg success earlier in the month, with a story I submitted to Digg actually reaching the front page. The discussion is well worth reading with some insight from Michel Fortin and others.

Linking to Wikipedia Technorati and Social Bookmark Sites Could Damage Your Google Search Results!

Here is an excerpt from the Google BlogSearch patent document that I discussed in depth earlier today:-

A link distribution of the blog document may be a negative indication of quality of the blog document. As disclosed above, some posts are created to increase the pagerank of a particular blog document. In some cases, a high percentage of all links from the posts or from the blog document all point to ether a single web page, or to a single external site. If the number of links to any single external site exceeds a threshold, this can be a negative indication of quality of the blog document.

Lets take a look at the type of links that people have on their blogs

Technorati Tag Links

Google in the same patent mentioned that tagging could be beneficial for the quality of your site, but what of the detrimental effects of linking through to Technorati with 10 links on every post? Maybe you are undoing all the good work of tagging your content.

Solution: Use local tagging, or just rely on lots of categories or labels to convey tag data – local tagging can be easily achieved using plugins such as Ultimate Tag Warrior if you use WordPress.

Wikipedia Links

Some people link through to Wikipedia it seems almost every other post. Despite Wikipedia being looked on by many as an authority site, such linking habits might be looked on as spammy.

Solution: Use nofollow on your links to Wikipedia

Social Bookmarking Links

This one is a biggie – some people have 10 or more social bookmarking links per page, and many of the most popular solutions don’t provide “safe” links. By safe I mean that the links serve no purpose because they go to an empty form, so in no way are they a pointer or vote on a particular piece of content. I am sure search engines probably discount them heavily, but they may still reduce the value of other links on your page, and may incur a ranking penalty due to being spammy.

I don’t use a normal version of the Sociable plugin, I use a hacked version. If you are using the Search Status plugin for Firefox, it is easy to tell, because you can see the pink boxes.

If you want to hack the Sociable plugin I wrote this guide a few months ago or just use my hacked version, now available.

Digg buttons such as “DiggThis” are not a problem, because they are normally Javascript. Some others use true dynamic linking using things like “onclick”.

Subscription Buttons

Yep more pink on this site, how about on yours?

It is ok to use a followable link to your feed, in fact that is recommended, even if you use and external service such as FeedBurner.

These buttons however link through to forms, not content. They should always be nofollowed.

Solution: Add rel=”nofollow” to the links

Conclusion

I am generous with my Google Juice, using Dofollow plugins for my comments, and link out to my readers often, but I hate pouring Google Juice down the drain.

For a long time my blogs have performed amazingly well with Google Blog Search. I always appear in the relevant results quickly, and the results I obtain have some reasonable longevity, even when I am not the original source of a story.

Considering how much competition I often have for certain search terms which everyone seems to be writing about because of common interest, I must have been doing a number of things right.

Bill Slawski of SEO By The Sea a few of days ago broke the news of Google’s Patent Application for Ranking Blog Documents.

SEO Round Table posted a synopsis lifted from the Cre8asite Forums that had been posted by Bill, and seems to be the easiest to understand.

I am going to do a little bit of mix and match here, and inject my own commentary but my interpretation of the patent is actually slightly different to those that I have read so far.

It should be noted I am working my way through the patent itself, and not recompiling the summaries of others.

Relevancy & Quality – Blog | Blogpost

It should first of all be noted that in the patent Google doesn’t differentiate between individual blog posts and whole blogs.

The phrase “blog document,” as used hereinafter, is to be broadly interpreted to include a blog, a blog post, or both a blog and a blog post. It will be appreciated that the techniques described herein are equally applicable to blogs and blog posts.

Later on in the patent, they also mention that feeds are also included within the documents that are compared and rated.

two distinct sets of data are used to determine a score of a blog (or blog post) in response to a search query–the topical relevance of the blog (or blog post) to the terms in the search query and the quality of the blog (or blog post), which is independent of the query terms. The quality of the blog (or blog post) may positively or negatively affect the score of the blog (or blog post)

Relevancy – this applies to the search term, thus Google will analyse the blog page, and they will also in some way determine the relevance to the whole blog.
Quality – this is irrespective of the search term, so think about factors from outside your niche

Google Blog Search – Positive Factors Affecting Search Quality | Relevancy

• Popularity of the blog document

A number of news aggregator sites (commonly called “news readers” or “feed readers”) exist where individuals can subscribe to a blog document (through its feed). Such aggregators store information describing how many individuals have subscribed to given blog documents. A blog document having a high number of subscriptions implies a higher quality for the blog document

This patent was first of all applied for 13th September 2005, with Google Blog Search launched 13 September 2005. At the time they were logically not basing this on numbers available for Google Reader subscribers. The Google Reader blog was launched October 21, 2005 with a post saying they had been up and running for 2 weeks.
Maybe there is a coincidence between the 2 events.

So which data were Google basing this part of their patent on? Some services such as Technorati and Bloglines do provide readership data, as does Feedburner, though most services report readership data as they are collecting new blog posts to a service like Feedburner, who aggregate the statistics.

It seems there might be some value is collecting Technorati favorites (my reciprocation policy might be well worth it) beyond limited bragging rights. Google of course through Google Reader now have access to lots of usage data, so maybe other sources will eventually be phased out.

• Implied popularity of the blog document

This implied popularity may be identified by, for example, examining the click stream of search results. For example, if a certain blog document is clicked more than other blog documents when the blog document appears in result sets, this may be an indication that the blog document is popular and, thus, a positive indicator of the quality of the blog document.

Click data from search results, possible from Google Toolbar users.

• Existence of the blog document in blogrolls

The existence of the blog document in blogrolls may be a positive indication of the quality of the blog document. It will be appreciated that blog documents often contain not only recent entries (i.e., posts), but also “blogrolls,” which are a dense collection of links to external sites (usually other blogs) in which the author/blogger is interested. A blogroll link to a blog document is an indication of popularity of that blog document, so aggregated blogroll links to a blog document can be counted and used to infer magnitude of popularity for the blog document.

Everything I have ever read has suggested that for normal search, blogroll links that are site wide carry diminishing value. Just because it is listed here as part of the calculation does not necessarily mean that everyone should start building up huge blogrolls… well unless they want to game Technorati and have a blog network.

• Existence of the blog document in a high quality blogroll

The existence of the blog document in a high quality blogroll may be a positive indication of the quality of the blog document. A high quality blogroll is a blogroll that links to well-known or trusted bloggers. Therefore, a high quality blogroll that also links to the blog document is a positive indicator of the quality of the blog document.

Another revelation, links on high quality pages are worth more than links on low quality pages.

Remember that “blog document” can mean both blog page and blog site.

Can blogroll just refer to a list of links on what is identified as a blog. Thus a column of links to related pages might also class as a blogroll, whether in the sidebar or below the content.
Thus a list of links to related documents on the same site could be looked on as a blogroll on a blog document.

Related links plugins are very powerful, especially if you also include them in content that gets syndicated by design, or by sploggers.

• Tagging of the blog document

Tagging of the blog document may be a positive indication of the quality of the blog document. Some existing sites allow users to add “tags” to (i.e., to “categorize”) a blog document. These custom categorizations are an indicator that an individual has evaluated the content of the blog document and determined that one or more categories appropriately describe its content, and as such are a positive indicator of the quality of the blog document.

Well some sites do allow you to tag in a meaningful way, maybe Google uses shared tags from Del.icio.us and other sites, but many of those use nofollow extensively.
It is my own belief that self tagging content heavily with plugins such as Ultimate Tag Warrior helps a huge amount. I have given lots of examples before, but more recent examples include

toolbar pagerank
google reader feedburner
feedburner google reader
compete toolbar
duplicate content supplemental results

Yes, I am just going down the inbound traffic results looking for likely candidates that rank well in both blog and normal search and aren’t totally obscure. These are subjects that sites in my niche have also talked about, with the keywords in the title, and which you would expect to rank higher than my own content.

This doesn’t just affect blogsearch, Google have been using it for some time with the main results as well.
Here are my observations regarding tagging from back in November, especially how they could relate to LSI calculations.

• References to the blog document by other sources

Wow revelation again, god links are worth having either to pages or blog.

• Pagerank of the blog document

Pagerank is still relevant, who knows for how long and how much.

It will be appreciated that other indicators may also be used.

What seems to be missing, at least at time of application?

• Domain age?
• Trustrank?
• Page Titles?
• URLs?
• Growth rate of link popularity

Plus lots more that also factor into it, but general search patents probably also cover blog search.

Google Blog Search – Negative Factors Affecting Search Relevancy | Quality

• Frequency of new posts on the blog document

The frequency at which new posts are added to the blog document may be a negative indication of the quality of that blog document. Feeds typically include only the most recent posts from a blog document. Spammers often generate new posts in spurts (i.e., many new posts appear within a short time period) or at predictable intervals (one post every 10 minutes, or a post every 3 hours at 32 minutes past the hour). Both behaviors are correlated with malicious intent and can be used to identify possible spammers. Therefore, if the frequency at which new posts are added to the blog document matches a predictable pattern, this may be a negative indication of the quality of the blog document.

Make sure there is some variation when you publish your content for the day, especially with future dated posts.
Most spamming tools are actually fairly sophisticated, thus I am not sure this measurement is very accurate. It most likely indicated a blogger who is very organised these days.

• The content of the posts in the blog document

The content of the posts in the blog document may be a negative indication of the quality of that blog document. A feed typically contains some or all of the content of several posts from a given blog document. The blog document itself also includes the content of the posts. Spammers may put one version of content into a feed to improve their ranking in search results, while putting a different version on their blog document (e.g., links to irrelevant ads). This mismatch (between feed and blog document) can, therefore, be a negative indication of the quality of the blog document.

This is actually a very significant and interestingly worded item. Google are stating that they are comparing the content of a feed with the content on your pages to ensure it matches.

Based upon this:-

• Don’t use a content spinner on your feeds to avoid duplicate content
• Allow Google to index your feeds
• If you use related links on your blog, make sure you use them in your feeds too
• Duplicate Content, especially in feeds

Also, in some instances, particular content may be duplicated in multiple posts in a blog document, resulting in multiple feeds containing the same content. Such duplication indicates the feed is low quality/spam and, thus, can be a negative indication of the quality of the blog document.

I can’t say I have noticed a problem having a lot of straggling RSS feeds on categories and tags.
This could also be referring to things like the large footer I have on each post, though I haven’t seen a problem with that either.

After the last toolbar pagerank update I spent some time studying Matt Cutts’ blog, and also looking at how pagerank was being transferred around my own site. Pagerank is only slightly useful as a guide, and only immediately after an update.
Rather than repeat myself, you can read about my organic garden approach to this site.

• Collective Intelligence

The words/phrases used in the posts of a blog document may also be a negative indication of the quality of that blog document. For example, from a collection of blog documents and feeds that evaluators rate as spam, a list of words and phrases (bigrams, trigrams, etc.) that appear frequently in spam may be extracted. If a blog document contains a high percentage of words or phrases from the list, this can be a negative indication of quality of the blog document.

Google invest a lot of research analysing spam, detecting various word matching patterns, and use that to identify other documents.

• A size of the posts in the blog document

The size of the posts in a blog document may be a negative indication of quality of the blog document. Many automated post generators create numerous posts of identical or very similar length. As a result, the distribution of post sizes can be used as a reliable measure of spamminess. When a blog document includes numerous posts of identical or very similar length, this may be a negative indication of quality of the blog document.

This might be of special interest to those that use out-sourcing for articles, you need to ensure the article length changes.

• A link distribution of the blog document

A link distribution of the blog document may be a negative indication of quality of the blog document. As disclosed above, some posts are created to increase the pagerank of a particular blog document. In some cases, a high percentage of all links from the posts or from the blog document all point to ether a single web page, or to a single external site. If the number of links to any single external site exceeds a threshold, this can be a negative indication of quality of the blog document.

In some ways this debunks the benefits of blogrolls mentioned as a benefit, but as previously quoted, Google are using blog document in multiple context, and comparing the context, thus it could just refer to multiple spam links always pointing to a single domain within the content.

• The presence of ads in the blog document

The presence of ads in the blog document may be a negative indication of quality of the blog document. If a blog document contains a large number of ads, this may be a negative indication of the quality of the blog document.

Remember this is just a patent, and Google recently relaxed the rules about having ads from other networks along with Adsense. As long as a page is of a reasonable size to support the adverts, I don’t think there is a problem. If you just have a heading and 5 words, with 10 advertising blocks, you might want to add a few more words.

However they go on to say this

Moreover, blog documents typically contain three types of content: the content of recent posts, a blogroll, and blog metadata (e.g., author profile information and/or other information pertinent to the blog document or its author). Ads, if present, typically appear within the blog metadata section or near the blogroll. The presence of ads in the recent posts part of a blog document may be a negative indication of the quality of the blog document.

Thus if you are using blocks in the content for all your ads, you might not rank as well, especially if you use multiple networks. You can probably get away with 3 in the content, or maybe 1 or 2 per post.

• It will be appreciated that other indicators may also be used

Conclusion

The feed stats information is very useful, and looking at the timing, my conclusion is that Google might have been using Bloglines and Technorati Favorites data, with Google Reader in its infancy, or maybe though less likely, when blog search was introduced, they weren’t using that part of the patent

For me the most significant information was tagging, but just linking though to Technorati with your tags isn’t a great idea.

Remember that Google have their own blogging system, and they have archives and labels, and they are not going to create a system to generate duplicate content and then penalise you for it. Google wouldn’t have added such a system unless they intended to benefit from the enhanced data.

You don’t have to build your blogs in a 1990s era tree like structure to rank well.

Subscribers and Technorati Favorites may help you rank.