However yesterday I read a post ranking cloud computing vendors based on mind share using a points system based on Google ranking for the term [cloud computing]

The winner was Rackspace, but we have some idea how they achieved that

But I was more interested in the result for Dell

First off they are doing some funny redirects so the displyed URL in the snippet
www.dell.com/cloudcomputing
Redirects to their cloud computing page on a subdomain which I assume is some kind of CDN

http://content.dell.com/us/en/enterprise/cloud-computing.aspx?c=us&cs=555&l=en&s=biz

Notice that there is only a URL displayed in the snippet and not an excerpt from the page or meta description. Google is just displaying the page as a reference, but for some reason couldn’t crawl the page.

Now before I show you what they have done wrong I did try to contact Dell about this yesterday. I sent a tweet to the guy who maintains at least one of their social media profiles, and since this tweet he has been tweeting on his account so I assume he has had a chance to read the tweet I sent.

I didn’t receive a response, and as this is a significant problem I am blogging about it, mainly because Google seemed to be doing something contrary to what I previously understood (they weren’t if you read more) and have seen written by authoritative sources such as Vanessa Fox.

Here is the robots.txt of that Dell subdomain

The page concerned us under /us/en/enterprise/ so should be controlled by this directive.
Allow: /us/en/enterprise/

As Vanessa states

If conflicts exist in the file, the robot obeys the longest (and therefore generally more specific) line.

So somthing is going wrong in between the URL displayed in the search results, and the final landing page.

Server Header Analysis

Search result
>> [302] http://www.dell.com/cloudcomputing
>> [302] http://content.dell.com/cloud-computing.aspx?c=us&cs=555&l=en&s=biz
>> [301] /us/en/enterprise/cloud-computing.aspx?c=us&cs=555&l=en&s=biz

Problem – Robots.txt Disallow Match

The inbetween URL redirect is blocked by robots.txt

http://content.dell.com/cloud-computing.aspx?c=us&cs=555&l=en&s=biz

It is actually matched by
Allow: http://content.dell.com/
Disallow: /

Google has decided to take the most restrictive version, to disallow the URL.

I haven’t checked how many links are pointing to that URL both from internal and external sources, but I would guess it is a fair amount and I am assuming at this time that is a page they would want to rank as in their line of business it is probably quite a competitive term.

At a guess without that one URL blocked by robots.txt, they would have been much higher on the comparison charts and ranking higher for cloud computing – most likely that is a 1st page result thrown away by robots.txt

I haven’t checked how many other key landing pages are blocked in a similar manner – I bet it is quite a few

This is just chalking up one more reason why robots.txt should be avoided as it is so easy to mess things up.

p.s. My niece is 18 in 2 months & I am sure she would love a Dell

For years I have been reading absurd suggestions that Google’s Blogger / Blogspot service is SEO friendly. At one time this was semi-true, and it is even possible with the current incarnation. However if you use the service “as intended”, without expert knowledge, you might find yourself up a creek without a paddle.

I am writing this on a Sunday, thus I am going to avoid vulgarities, cussing, and even an attention grabbing headline. The Google engineers at Blogger deserve that and more. Blogspot currently stinks for SEO.

Google Webmaster Guidelines Mess Up

This is probably the root cause of the Blogger problems

Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

Blogspot Doesn’t Have Categories

There was a historical lacking feature in Blogspot, no categories, and users wanted them.

So Blogger introduced labels, but they unfortunately followed the advice from their own Webmaster Guidelines, such as this example from Blogger’s own Blogger Buzz blog.

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search

Sitemap: http://buzz.blogger.com/feeds/posts/default?orderby=updated

That causes label pages such as this one on the Google Webmaster blog not to be crawled

http://googlewebmastercentral.blogspot.com/search/label/webmaster%20tools

The links to those tag pages don’t have nofollow, but Google can’t crawl them to read what is on the page, but pages blocked using robots.txt still accumulate PageRank.

Blogger blocks labels with Robots.txt

That creates “hanging or dangling pages” and the PageRank gets shared out between all the pages in the internet

There are easy solutions to this:-

Best solution

• Have excerpts on the label pages
• Possibly noindex the label pages with either meta noindex or robots.txt noindex
• Add nofollow to all external links on the label pages, both in the content and sidebar

Stop-gap solution

• Nofollow label links from content – they still wouldn’t give any deep indexing benefit, but at least they wouldn’t throw PageRank away.

Note: Those solutions are for blogger developers to implement instead of using robots.txt disallow, not for Blogger users. Sorry Blogger users, you have been at a significant disadvantage for years unless you can find a smart (viable) alternative to creating categories by hand.

Blogspot Indexing Problems

With labels being blocked by robots.txt, some but not all Blogspot blogs have major indexing problems.

As an example, it doesn’t seem like Google passes any juice through the dropdown menus used on the Webmaster Central blog.

Dropdown Menus On Blogspot Could Cause Indexing Problems

Of couse not every Blogspot publisher displays date based archives, and even the traditional “older posts / newer posts” navigation ends up being blocked by the robots.txt file.

As an example this link from Rick Klau’s blog (A Blogger Product Manager)

http://tins.rklau.com/search?updated-max=2009-05-04T11%3A53%3A00-07%3A00&max-results=7

You can see by the lack of snippet and cache that this is blocked by robots.txt

Blogger Older Posts Navigation Blocked With Robots.txt

Fortunately for Rick he doesn’t allow his labels and pagination links to be the only method for Google to crawl his content, plus he has a huge amount of “Google Juice” to play with.

This is actually the best navigation implementation I have seen on any Blogspot blog, as it has allowed all his content to be indexed, or certainly the vast majority.

Blogger Navigation That Works

This is still “ball linking” rather than a structured appraoch, but for Blogspot it is about all you could hope to achieve.

By having this massive navigation on every page, effectively turning every page into a sitemap, there are so many link on page that a few external or internal label links don’t really matter.

This isn’t the case with the majority of Blogger blogs, and blocking with robots.txt is an abomination.

At the same time Rick has been trying to persuade people to stop using the old, unreliable Blogspot system which allowed you to publish to your own domain, suggesting that there are no problems.

How Hard Is It To Leave Blogger?

Whilst people talk about it being easy to leave Blogger to another platform, the importer has been bugged for a long time – Blogger strips stopwords from generated URLs, WordPress doesn’t.

Then there is only what I would term a “Interstitial Defamation Wall”

Redirects from Blogspot to a custom domain now hosted externally get slapped with an interstitial warning.

See for yourself http://niche-website.blogspot.com

I had already left Blogspot before Google decided to finally provide custom domain support. I had set up a complicated script with meta redirects, and I even regained my rankings and wrote about it.

When custom domains became available, I decided that it would be best to assign andybeard.eu to my blogspot account, and then point it at WordPress so that visitors and search spiders saw an immediate 301 redirect.

Google when you set this up are very reassuring

Blogger Custom Domain Setup

It seems this is only true if you stay with Blogger forever. I should also point out that during my testing I discovered that I could no longer assign andybeard.eu, and I was forced to use www.andybeard.eu

Links & Rankings

The main thing that ties any Blogger to using Blogspot are incoming links and rankings. When I moved I probably had 24,000 links to my old Blogspot, depending on which reporting you believed.

Even a couple of weeks ago Yahoo was reporting ~10K , though it seemed a lot less today. Google in Webmaster tools only reports 100 or so, though they may have removed loads due to the redirects.

I don’t care that much, building up links takes time but isn’t something that can’t be fairly easily replaced over time, at least for me.

However for any potential blogging service provider (and honestly anyone running WordPress might soon be able to offer blog hosting within 3-6 months if WordPress core merges with WPMU) this is a serious barrier to overcome.

If you use Blogspot, but with a custom domain from the beginning, this isn’t a problem, but for everyone else, it might be a deciding factor to stay.

I could switch back to the delayed meta refresh method, but I can’t see why I should have to – Google should never have inserted such a message between me and my visitors.

I am looking to clean up some old articles because there is too much SEO junk on the web, but deciding how to handle my old “How to move from Blogspot to WordPress” articles are a major problem.

• I am adamant that people really should still leave Blogger for their own good, if they value their online business
• I am not in a position to advise on best practice, and the most suitable method is messed up.

As far as I am concerned this interstitial isn’t passing any juice through to Andybeard.eu, though there is a YES/No link, but it uses onclick, but returns false, but the URL is not constructed… ask Sebastian if you want a definitive answer.

I am not going to call this a perfect example of a reconsideration request, but I decided that it was better to be 100% honest about my thought process for both now and in the future, because whilst I am now blocking paid reviews with robots.txt, there are so many things still not specified within the webmaster guidelines that it is a potential minefield, especially for someone who has previously been the target of a manual penalty.

Here is exactly what I sent to the Google webmaster team.

I honestly still believe I didn’t break the spirit of the webmaster guidelines, the webmaster guidelines as most frequently described by Google employees on official duties in regards to paid links and reviews, and even the “letter of the law”.

I have a very high rejection rate on paid reviews, approaching 80% – that shows editorial discretion far beyond many if not all paid directories.
Content was always highly targeted to my audience

I retained editorial control of links – in all my blogging I give good search engine friendly links that are descriptive of the target – such practice is effectively law in the UK and Europe, though who is responsible for regulation isn’t certain.

Almost a year ago, when Google first made it possible to report a site for paid links, I reported myself with a request for clarification – at that time how my reviews were regarded by Google was not clearly specified, in many ways it still isn’t.

Adam Lasnik previously suggested multiple times that sites which were predominately high quality content were not your target for penalties – paid reviews content currently represents less than 3% of my original content – just 9 reviews

I strive to provide an example of paid review content that is benefiting readers in general, and has a reason to be indexed and counted as editorial content.
The compensation I receive is more a token gesture, like a box of chocolates to say thanks, as the time I spend on them means I would earn more flipping burgers in McDs – how could that class as paid links? It doesn’t even cover my time.

Penalties have not been handed out evenly, I know Googlers read blogs that have written paid reviews where the links were not blocked in any way, and have even commented on the specific reviews. Those sites remain unpunished.

Googlers continually promote Google services from their private blogs, and certainly gain financial compensation from increased stock prices. Just today Matt wrote the following post without a specific disclaimer that he is a Google employee.
http://www.mattcutts.com/blog/adding-new-features-to-google/
Surely such posts should have a nofollow to Google – it has purely promotional (though helpful) intent.

All that being said, I had no intention to break the Google guidelines, and if what is required for the Google Toolbar to truthfully depict the authority of my website is for my editorial links in paid reviews to be blocked from Google in some way, I am going to comply.

All permalinks to my paid reviews are now blocked using robots.txt – I have checked that this is the case within webmaster tools

Where excerpts of my articles appear on duplicate content pages, all links are nofollowed, such as on tags pages http://andybeard.eu/tag/volusion_review

Ultimately the Google search results will be poorer quality, because a good paid review is a better result than an article based upon a press release, or a SEO optimized press release itself. I would argue that the links are of higher value as well.

There are still going to be cases where what I write will be within the grey area not defined by your webmaster guidelines.

I earn money from Google through Adsense – should I nofollow every link to Google and Google services now?

I have paid advertisers, many are purchased by my regular readers or services I frequently write about in an editorial manner. Do I now need to nofollow every past current and future link to them because I have accepted a small amount of money for display advertising?

Do I need to nofollow affiliate links? I can’t see an automatic way that Google can tell the difference between an affiliate link, and a paid link that has a tracking parameter.
It is somewhat strange that Google has provided help for merchants in cleaning up search results with affiliate links using redirects, and thus gaining an SEO benefit from them. Affiliate links very frequently are not editorial endorsements.

My content gets syndicated often on authority sites such as Webpronews and Searchnewz, and many lesser sites – I trust that if they choose to publish my syndicated content as an editorial decision, that the fact that content was originally a paid review is no longer a problem.

Best regards

Andy Beard

Before filing it I checked that the changes I previously made had taken effect and the pages were actually blocked.

I also made changes to my disclosure policy to make sure that it was 100% clear to anyone from Google performing a manual inspection that all links in paid reviews from my domain will not affect search results.

This was important based on the discussion regarding Donna’s reconsideration request.

Something For The Naysayers

There were people who for some reason thought that blocking my paid reviews using robots.txt would for some reason be extremely harmful to my search traffic.
It is true that a document that can’t be indexed cannot rank for long-tail phrases within it, but pages blocked with robots.txt can still rank in Google.

That is just a snapshot of a SERP, it is certainly bouncing around a little and the position is changing daily.

That being said, that page was previously ranking 12th, and whilst it isn’t a high traffic term, it is quite competitive with lots of theme and plugin authors also attracting lots of links.
I haven’t done as much as I could do to promote the page because it is a paid review.

In addition I have flattened my internal linking structure over the last week – frequent detractors obviously wouldn’t look on that as a major search ranking factor.

This article does not reference Google Toolbar PageRank in any way

First of all you are going to need to do a little homework.

Eric Enge interview with Matt Cutts

The Eric Enge interview with Matt Cutts was truly exceptional and revealed a number of gotchas that for some reason continue to be circulated.

Key takeaways

Matt Cutts: … Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results.

Matt Cutts: … So, with robots.txt for good reasons we’ve shown the reference even if we can’t crawl it, whereas if we crawl a page and find a Meta tag that says NoIndex, we won’t even return that page. For better or for worse that’s the decision that we’ve made. I believe Yahoo and Microsoft might handle NoIndex slightly differently which is little unfortunate, but everybody gets to choose how they want to handle different tags.

Eric Enge: Can a NoIndex page accumulate PageRank?

Matt Cutts: A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page.

Eric Enge: So, it can accumulate and pass PageRank.

Matt Cutts: Right, and it will still accumulate PageRank, but it won’t be showing in our Index. So, I wouldn’t make a NoIndex page that itself is a dead end. You can make a NoIndex page that has links to lots of other pages.

For example you might want to have a master Sitemap page and for whatever reason NoIndex that, but then have links to all your sub Sitemaps.

I have just provided a couple of highlights, I am not attempting to replace a need for visiting the site I am citing. This is something I hate seeing, when people take other people’s content and repurpose it, thus making the original article worthless.
There are a few other gotchas in there, I suggest you read it 2 or 3 times to really understand what was said, and what wasn’t said.

Dangling Pages

One of the best descriptions of dangling pages is on the Webworkshop site, though they are assuming that links are totally taken out of the equation based on what they quote from the PageRank paper.

“Dangling links are simply links that point to any page with no outgoing links. They affect the model because it is not clear where their weight should be distributed, and there are a large number of them. Often these dangling links are simply pages that we have not downloaded yet……….Because dangling links do not affect the ranking of any other page directly, we simply remove them from the system until all the PageRanks are calculated. After all the PageRanks are calculated they can be added back in without affecting things significantly.” – extract from the original PageRank paper by Google’s founders, Sergey Brin and Lawrence Page.

Alternate interpretation

This is just an aside, as the amount of juice lost to dangling pages currently is hard to determine, and could be handled differently

They are assuming that if page A links to 6 other pages, 5 of them being dangling links, then the website will be treated as only having 2 pages until the end of the calculation.

Whilst I haven’t delved into the maths (and probably couldn’t through lack of information and lack of knowledge), it also seems to me that at the time the pages are taken out of the cyclic calculation, a percentage of the link value can still be taken with them.

Thus though the site for cyclic calculations will be just 2 pages, the link from A to B might only transfer 1/6 of the juice on each cycle.

At the time the original paper was written, Google only had a small proportion of the web indexed due to hardware and operating system restraints.
In modern times they have a lot more indexed, thus a more complex way of handling dangling pages could be possible.

More food for thought, a link to a page that is considered supplemental could be treated as a full link or as a link to a dangling page, or some other variant.

Even more food for thought, a site with multiple interlinked pages with no external links at all could be looked on as a “dangling site”.

Ultimately what is important is that dangling pages are a juice leak, though it is difficult to determine exactly how much

Additional Research On Link Juice Flow

I have referenced these works before, and I am just going to keep on referring people to them.

• SEOFastStart by Dan Thies – a good introduction to SEO, and also introduces the ideas of controlling juice around a website – no email signup required
• Revenge of the Mininet by Michael Campbell – a timeless classic as long as PageRank continues to be important – the download page isn’t hidden if you really don’t want to sign up to Michael’s mailing list, but I have been on his list for years.
• Dynamic Linking by Leslie Rhode – A bonus that comes with Revenge of the Mininet

I mentioned these is a comment on SEOmoz recently in a discussion on PageRank, and for some reason my comment received just 2 up votes and one down vote.

I don’t gain in any material way from promoting these free ebooks, though I might gain some goodwill. The main reason I link to them is because they are a superb resource, and it saves me countless hours writing beginners material.

OK, On to some debunking

Blocking Pages With Robots.txt Creates Dangling Pages On The First Tier

In the quoted paragraph above, Matt clearly states that pages blocked with Robots.txt still accumulate juice from the links they receive.

Those pages don’t have any external 2nd tier links that are visible to a ‘bot, thus they are dangling pages.

How much juice they leak depends on how Google currently factor in dangling pages, but Matt himself suggests not to create dangling pages.

If you read any SEO Guide that suggests that the ultimate cure for duplicate content is to block it with robots.txt, I suggest you might want to question the author about dangling pages.

Meta NoIndex Follow Duplicate Content

This is a better solution than using Robots.txt, because it doesn’t create dangling pages. Links on a duplicate content page are still followed, however both internal and external links are followed and thus are leaks, often multiple leaks for the same piece of content when using CMS systems such as WordPress which create site-wide links in the sidebar when using poorly designed themes, plugins, and especially WordPress Widgets.

If you read an article suggesting using Meta Noindex Follow, ask the author how they are controlling external links on duplicate content pages.

Meta NoIndex Nofollow Duplicate Content

If you use Meta Noindex Nofollow, whilst this is handled slightly differently by Google to Robots.txt, as the page won’t appear in search results, it is still a page accumulating Google Juice if you link to it, another dangling page or node.
Second tier leaks from the page won’t leak, but the page as a whole will leak depending on how Google are currently handling dangling pages.

I don’t see people recommending this frequently, but as with Robots.txt, ask the author about dangling pages.

Dynamic Linking & rel=”nofollow”

Extensive use of Nofollow and other forms of dynamic linking are the only way to effectively prevent duplicate content pages in some way having a effect on your internal linking structure and juice flow. The Wikipedia page on Nofollow really isn’t correct.

The Dangling Sales Page

To finish I want to give you an example of how a sales page that previously might have benefited from lots of links can easily be turned into a dangling page and effectively discounted from cyclic PageRank calculations.

Sales pages started off just as a single page with no links:-

Despite all the links coming to the site from external sources, this website is a dangling page, thus excluded from iterative PageRank calculations. It might still benefit from anchor text and other factors, but it effectively is not part of Google’s global mesh and passes on no influence.

Add Legal Paperwork And Reciprocal Links Directory:-

A much more structured site, and whilst it gains some benefit from reciprocating links there are 2 factors that are almost universally overlooked.

1. No Longer A Dangling Page – because the site now has external links, it is valid as part of the global ranking calculations. Other pages as mentioned above were previously stating that the amount of juice passed to dangling pages was minimal, so this could be potentially a huge boost.
2. More Pages Indexed – it is only a few pages, but with PageRank it is often not just how much juice you have flowing into a site, but what you do with it.

The reciprocal low quality links might not have had a huge amount of value compared to the benefit of being a member of the “iteration club” and having a few more pages indexed.

Add a link to the designer

Some early single page sales letters were not dangling pages, but didn’t benefit from any internal iterations, and acted as a conduit of juice to their web design firm.

The Danger of Using Nofollow or Robots.txt on Unimportant Pages

I have actually seen this on a few sites:-

• Reciprocal Link Directory Removed
• Link to web designer removed
• Nofollow added to legal papers that are looked on as being unimportant

Such a website is now out of the iteration club, it is a dangling page as it is no longer voting on other pages.

My Own Gotcha

I mentioned that this catches me out as well.

A while ago I wrote an article about linking to Technorati being a problem. It might still be true, but the amount of juice lost through such links might also be lower than I thought, due to Technorati using meta nofollow on every page. Technorati tag pages are themselves dangling pages with no external links.

Wikipedia and Digg on the other hand are not dangling pages. They still have external links to other sites, and thus any links to them are part of iterative calculations.

I would still say it is best to have tags pointing to your own domain tag pages, and to use nofollow on links to Wikipedia and Digg, though with Digg I suggest that is only on links to submission pages which contain no content.

Stumbleupon is also tricky – there are no external links from individual pages, but there is extensive internal linking.

With Digg and Stumbleupon, profiles rank extremely well, so you can use them for reputation management even if you get no juice direct from the profile.

I think I was the first to describe Wikipedia as a black hole of link equity, explained why you should nofollow Wikipedia extensively, and was one of the first to promote Ken’s Nofollow Wikipedia plugin.

You would have thought in 10 months they would have come up with an alternative to using nofollow on all those out-bound links.

They do however link out to a few trusted sites without nofollow, from just a few pages. I suppose Google does still allow them to be part of their iterative calculations.

Another Own Gotcha

This isn’t 100% something I can fix. I have suggested people use robots.txt on certain sites knowing it wasn’t the perfect solution.

You might notice on this site I don’t use an extensive robots.txt, and the design of my site structure is deliberate, but then at the same time I use nofollow with lots of custom theme modifications, and should use it a lot more.

Eventually I will come up with solutions to make things a little easier.

Tools In The Wrong Hands Can Be Dangerous

Using Robots.txt and Meta Noindex, Follow as a cure for duplicate content is a SEO bodge job or SEO bandaid. It may offer some benefits depending on how dangling pages are being handled, but is certainly not an ideal solution due to the amount of leaks that typically remain or dangling pages that are created.