I can’t even pinpoint within my data as to when this foolish or unfortunate incident occured because well… just look at the data.

That is nigh on 5 years of Google Blogsearch referral data, though because of the long time range it is listed as sampled data. There may be some traffic sources I have missed – variations of url depending on how Google were displaying blogsearch pages, but that is referrals from http://blogsearch.google.com

• In the past when I have checked at various times (though admittedly it has been a while) I had ranked well in blogsearch.
• There have always been so few referrals that I have more or less ignored Blogsearch
• The only blogsearch I have used are for links to me – in the WordPress interface, and occasionally to grab more results than WordPress displays – it has always been unreliable
• Blogsearch picks up links from all kinds of things
  • blogroll links
  • comments
  • if you send a pingback and a blog displays them, it will come up in blogsearch too

Because of all of these factors I had always assumed that the idea of using noindex on a feed of any kind was to prevent that feed appearing in Google’s primary organic results.

A pretty Feedburner feed isn’t a terrible landing page, but it is possble to do better. I have even written about using my feedburner URL when leaving blog comments in the past, as in some ways it immediately signals you want people to subscribe more so than linking to a blog.

Other RSS search engines were indexing my feed content – Technorati, Blogcatalog, Icerocket – my feeds were being read by my readers, picked up by various Twitter robots etc.

And of course my content remained indexed in Google’s primary organic index.

But then a few days ago I was browsing a little and looking for additional sources for a story followon, and noticed I wasn’t listed for previous coverage. I hadn’t been specific in the title that I was related… but there wasn’t a lot of competition.

Then I discovered this:-

My first thought for 5 minutes was somehow for some crazy reason I had become penalized in Google Blogsearch – then I rationalized it in thinking it must be something to do with noindex settings in Feedburner.

You see I had never equated noindex with a blog search engine – every other blog search engine which sent me traffic was still picking up my content and sending me traffic.

Google Indexing RSS Feeds

There is still a very real need for a way to tell Google…

“Hey Google, this is my RSS feed – you can index it for Google Blog Search, but I don’t want it to appear in the organic search results.”

3 years ago Google were saying they were working to remove RSS feeds from organic search.
3 years later feeds from Feedburner are still appearing in organic search results.

http://feeds.feedburner.com all the results seem to have been removed
http://feeds2.feedburner.com there still seems to be plenty of feeds within the search results

Information About & Help With Feedburner Since Google Acquisition

On a scale of 1 to 10 Google Feedburner Support gets a 2 – it is a free service, Google monetize it providing Adsense for feeds, but don’t expect anyone to answer support queries in the Google groups from Feedburner.
Documentation is sparse – hardly updated in the over 3 years since Google bought Feedburner… but then there haven’t been too many visible changes other than adding Adsense. I am sure there have been changes to help with scaling, especially how it eventually was made easier to integrate with Blogspot, but very little for anyone else.

Feedburner Noindex Controls

So this I believe is the culprit

This is the code that gets added to the RSS feed.

<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" /><meta xmlns="http://pipes.yahoo.com" name="pipes" content="noprocess" />

That data is still not transferred to feed items that are shared within Google Reader or feeds such as tags created there – which can get fed to other places and indexed.

The left hand doesn’t know what the right hand is doing

I have explained my whoopsie, but somewhere in the Googleplex they are a little confused over what they are doing as well.

Blogsearch isn’t the only search for my Blog posts

For instance there is Google Buzz

Now remember – Google is treating the noindex on my RSS feed as being an instruction to not include my content in Google BlogSearch…. so you would expect that instruction to be universal for the RSS content.

Those were taken from the PUBLIC timeline of Buzz. That is content that Google isn’t indexing on Blogsearch due to a noindex in the XML.

I also have my full content being fed into Facebook and being indexed and made searchable within Facebook, but at least that is my choice.

The only way to prevent content being shared and indexed is currently to block Google Reader from accessing feeds. I have been trying for over 4 years to get Google to introduce more publisher controls for sharing… as it would be easy to share private content from Google Reader by mistake… with Pubsubhubbub it can be broadcast by mistake to your 1000s of Buzz subscribers instantly.
This is possibly why Google have never introduced support for http authentication.

With their current stance for sharing freedoms, it doesn’t make sense for them to treat the current xml declarations as an instruction not to index the content in Blogsearch, as the content is in Buzz anyway. It should be treated as just a noindex for the page.
Alternatively they should add support for x-header noindex, then noindex in the XML would be for search engines, and it should travel with each content item, even to Buzz, possibly with no way to share the content.

I can understand why Matt Cutts might want to change what noindex does, it is not just Koreans making occasional mistakes, for instance I just noticed the whole WebProNews video blog is currently noindex nofollow. I am sure that is a mistake, it is easy to make in WordPress… just one click and save.

All in one plugins are dangerous if you don’t know what you are trying to achieve

Rel-nofollow, meta instructions and robots.txt are just tools.

Just because a tool is available to use in your toolbox, doesn’t mean you should use it. You don’t always need to use a big hammer to repair a TV set though sometimes a big hammer just isn’t big enough ;)

Inclusive PageRank Sculpting

Whilst I agree with Michael that nofollow has a purpose and I use it as a tool to achieve desired results, Shari raises some good points.

The Search Engine Land blog doesn’t use nofollow on links, but the “information architecture” is sculpted with a very flat profile. SEL is an information resource, and all pages are given almost equal emphasis.
That may not be true of a niche website, e-commerce site, etc – this is something that has to be determined on a case by case basis.

If you are looking at WordPress SEO in a competitive niche, for specific keywords, having a specialist toolset available is an advantage.
That page is blocked by robots.txt, but still ranks highly for reasonably competitive terms based upon anchor text within plenty of editorial links, and internal linking structure, domain authority etc.

Pages blocked by Robots.txt still accumulate Google Juice.

But you shouldn’t give a baby razor blades to play with, let alone a chain saw.

Less Important Pages Can Be Your Quarterbacks

My sitelinks are currently a total mess, and haven’t been updated since November. I have so many links on my front page that Google has a hard time to determine which pages are most important, and some internal pages have attracted a fair number of external links.

On many sites, a contact form, privacy policy, advertising page etc might be important to appear on your sitelinks, but less important in standard search results.

Simple Site Structure

Here we have a simple site structure, with 14 landing pages linked to from the front page. Of these pages, we have determined that 6 are unimportant, and 8 we would ideally like to appear in sitelinks.

The following are just a few examples of how we could arrange the linking structure.

I should point out the following:-

• It is much more complex than these simple diagrams because I haven’t included any 3rd tier (or deeper) pages
• I haven’t included any home links or links from a 3rd 4th etc tier to higher tiers
• Iterative calculations need to be thought about
• There are no leaks

Keep Them Out Of The Index

In this example the pages are so unimportant we want to keep them out of the index, and prevent them soaking up any Google Juice.

We nofollow links to them, and use meta noindex follow on the pages – this keeps them out of the index even if someone else links to them, but allows juice from those links to flow to other pages.
In this situation we should also ensure that there are no external links on the page using plugins such as my nofollow those dupes.

Not So Important Pages

Here we are thinking about usability – we still want those unimportant pages appearing in a site search or a more specific long tail search, but they are not key search terms.
Maybe some of these pages have an effect on site quality.

We nofollow links from the homepage, and maybe from the sidebar or footer throughout the site, but have a live link from the HTML sitemap.

When You Have Multiple Levels Of Importance

In this example we have 3 tiers of importance

• Pages we don’t want in the index or receiving any juice
• Pages that are important for navigation, but should receive less juice
• Primary pillar content aimed at competitive keywords

Our least important pages are still linked heavily, but have no juice being passed to them due to nofollow, and have noindex follow to keep them out of the index, but passing juice if they happen to receive a link.

Our low priority pages are gaining links from the HTML sitemap

Our most important pages receive juice from the home page, and possibly site-wide links.

Nofollow Is Not Required

Nofollow is not a requirement to enhance the flow of Google juice around a site, but it certainly helps.

In this particular case, we have some less important pages that are receiving a lot of jucie, maybe with site-wide links, not just from the home page.

However those pages are not as important as other pages on the site, and would not be our preference to appear as sitelinks.

We allow our unimportant pages to receive the blessing of a front page appearance, but the only links from those unimportant pages are to our most important pages. Juice flows straight through like a 100% efficient conduit.

If we only link to those unimportant pages from the front page, the total amount of juice they receive will be greatly reduced.

No Robots.txt

In these examples I didn’t use robots.txt once.

Robots.txt can be used strategically and in many cases is easier to implement than selective nofollow on links, or noindex follow on pages, but that doesn’t make using it “best practice”.

So Which Method is Best?

None of them, all of them (I just know I will get that question)

SEO is art, you can teach someone to hold a paint brush and all about perspective, but a true masterpiece requires creative talent and a lot of practice.

With my SEO articles I try to go a little further than just showing you how to hold a fishing rod or paint brush