At least if you look at it from a search engine perspective, who should assume that every visitor isn’t a member of the site they are referencing in the search engine.

It should also be understood that both Facebook & Twitter are bursting at the seams with former Google engineers & execs – they can’t claim they were unaware of what Google is looking for from content owners on the web, webmaster guidelines etc.

Twitter

You can’t look at the Google cache and see exactly what Google sees, because they do some sneaky redirects which are very akin to cloaking.

I have written about this before.

Video Exclusive: Has Google Given Twitter a Cloaking Penalty?

This is what Google sees based upon the preview

The little piece of text at the top of the page is what amounts to your profile… you can’t count the background image if any because it can’t be read by Googlebot unless it works really hard using OCR, and certainly can’t be read by people with disabilities.
The links within the content of the page are mostly nofollow, and the links in the sidebar get blocked by robots.txt.
The link at the bottom of the page to access more content… which may be of interest to search is also blocked by robots.txt.

I am not the only one who has spent considerable time trying to get Twitter fixed. A great example is this post by Vanessa on Search Engine Land.
How Twitter’s Technical Infrastructure Issues Are Impacting Google Search Results

Facebook

Facebook is worse

There is nothing there of any real value… it isn’t the timeline a logged in user might see.

First Click Free

If you want to have some kind of membership wall for users, then Google have special arrangements where you are required to show content for the first click.

Cloaking

Google over the years have published lots of content about what they think of cloaking.

I can still think of a few cases where some kind of cloaking would be justified. As an example on uQast we serve RTMP video with flash and use javascript “cloaking” to provide mp4 for iPhone. We could even serve that video to Googlebot’s mobile crawler without breaking Google guidelines as “cloaking” to serve content to specific browsers is allowed. But we can’t serve Googlebot which crawls for the main search index something it understands, as the Google guidelines require you treat Google as a normal desktop user browsing from California in the USA.
So Googlebot is served flash based RTMP within the webmaster guidelines rather than something it might like to see which we would be quite happy to give it.

That doesn’t prevent Google sometimes (though rarely) indexing the mobile video by figuring out the javascript, but it would be so much easier to give them something they understand.

Google Isn’t Playing Fair

One area that Google isn’t necessarily playing fair is that I don’t seem to be able to view Google+ profile pages in their own cache, and they don’t give a preview of the page that Googlebot sees.

This is my Google+ Profile

You can normally search in Google for cache:https://plus.google.com/102279602913916787678/posts or any url to get a cached version of what the crawler sees.
It is possible for every site to tell Google and other search engines not to store a cached page, so Google are well within their rights not to do so… but it prevents comparrisons.

Compare
cache:andybeard.eu – brings up a cached result
cache:https://plus.google.com/102279602913916787678/posts – does not bring up a cached result, just a 404 error

FTC Complaint over Search Plus Your World

The blogoshere love a good witch hunt, but I can’t see that Google is treating Twitter or Facebook unfairly. Eric Schmidt was quite right about some of the nofollows, but there are bigger technical restrictions in place on crawling.

I actually quite like a Google profile as a default profile and identity on the web, but Google need to live up to the promise of salmon and make it a viable endpoint for all activity, or as an alternative use it for identity, and allow me to define my own default profile.. which if I choose might be Twitter or Facebook.
I can also understand why you wouldn’t undertake the complex engineering to make such flexibility possible for your first itteration, especially with partners who are unwilling to do something similar themselves.

Just ask Twitter how many content partners they now support on the new Twiter for embeds. (I wrote them a letter a year ago and never received a response)

This post ignores what a logged in and fully javascript supporting human might experience, but in many ways Google’s profiles whilst now having a social element for years have generously linked out to any other online destination of your choosing, and provided the necessary markup to claim them as being part of your personal social graph.

Update – Google Profiles Now Cached

Michael VanDeMar left a comment showing a way to get the cached page to show by including the https protocol at the beginning of the url to query.

However when I posted I had tried lots of different variations all resulting in a 404 error.

This unmodified link was previously bringing up a 404 error
cache:https://plus.google.com/102279602913916787678/posts

It now returns what appears to be a blank page – as Michael points out if you switch off the CSS in your browser you can see the complete cached landing page.

Click to view full size without CSS

This appears to be a recent change, though they still need to fix the canonical – the canonical changes as you navigate between tabs and between the first 2 urls on this list there is effectively a redirect loop with /posts claiming / is the canonical, but humans are redirected to /posts

https://plus.google.com/102279602913916787678/

https://plus.google.com/102279602913916787678/posts

https://plus.google.com/102279602913916787678/about

All the different URLs show all of the same content, so should set whichever canonical a human is redirected to which currently is /posts

Not Total Fix

It seems some other pages are still giving 404 errors – maybe due to all the funky redirects going in circles with the canonical on occasion (this query is with HTTPS)

If you have difficulty understanding the concept of canonical, it is just like Highlander… “There should be only one” page with the same content in Google’s index, especially on the same domain.

Watch this video for the details

To check the cached Google page you will have to switch off javascript using something like the web developer toolbar for Firefox.

Google is not being blocked as there is a cached page, but the cached page is not what is given to users, even when not logged in.

The pages are significantly different in structure.

Google have quite extensive webmaster guidelines including a special on on cloaking and sneaky javascript.

Just to qualify the value of this video – some people rightly or wrongly look on me as an SEO expert – I don’t proactively consult partially because I don’t believe my own opinions half the time and in the past I have spent a great deal of time trying to prove myself wrong.

Ultimately even Google engineers directly involved with webspam or working directly on Google’s algorithms don’t know every aspect of how Google’s algorithms might work in every eventuality, and some results or side effects may be unintended.

It could well be that Twitter left old Twitter in place because Google was having problems crawling the new primarily javascript Twitter. From my perspective the mistake was to do a funky javascript redirect and having a significantly different link graph, but there could easily be something I have overlooked.

Firstly I like linking to people who link to me, whether on the post they first wrote, or on the syndicated copy that now appears on SEOmoz, even when the name referenced is “Andy Beal”.

Google is going to have a hard time deciding which is duplicate content, and will probably pick the SEOmoz article because it is the domain with the most authority.

If you syndicate articles or blog posts, make sure they link back to the original version, whichever you consider original. I am not going to help Google, as I have linked to both.

Android vs Blackberry Smartphones

I probably know as much about smartphones as Matt Cutts does about… poodles (he is a cat lover)

I have a SIMM card with a 7 mbps connection, but purely as a backup or for when I am travelling around Poland and am somewhere I can’t get good wifi. The SIMM works in one of my wife’s cast off mobile phones in an emergency.

As detailed in the linked posts, Google gave away lots of Android mobile phones to developers. That is something I am very familiar with – I used to work in the games industry and among other things handled relationships with all the PC Manufacturers. AMD, Intel, Creative Labs, Nvidia, Matrox,  etc etc.

Even though NDAs have now expired (I think the longest was Intel’s at 5 years) I am not going to go into specific details but here are the challenges.

• Developers had to create custom code to support specific features – this could take days, weeks even months.
• The testing teams would have to text code in a matrix, combining various processors with graphics and sound cards
• The support teams would have to create documentation for each possible platform and potential conflicts

In those days we were working with multiple standards, processors had lots of proprietary 3D functions, graphics cards not only had different features, but also different graphics libraries to access them, 3DFX, OpenGL and DirectX, and even sound cards had different features and sound libraries.

Some might look on it as a lot of back scratching, but it was a symbiotic relationship – it probably still is.

Developers had early access to hardware, sometimes months, even a whole year in advance. Different terms were subject to negotiation, status etc.

In exchange there were lots of cross-marketing possibilities, certainly linking happened, but also branding on boxes, adverts, possible lucrative OEM deals etc.

Whilst this might seem to favor the larger development studios, and it did in some ways, ultimately small development studios, if they got on board could certainly gain a “leg up” from the hardware guys, and this is something I was very active to encourage.

Thus Google giving away a few hundred, even a few 1000 mobile phones is barely a grain of sand compared to what is given out behind the scenes.

Google I/O Was Press

From what I can see, there were tons of press representitives at Google I/O, they received tons of coverage from notable tech blogs.

Press have always received free samples of hardware, or at least most have, though many publications have rules about keeping the “gear”, auction it off for charity, give it away as prizes etc.

In doing so that can help them remain impartial because they are not keeping the item.

Paid Links

The paid links saga of 2007 didn’t really clear anything up and effectively swept issues under the table, with the untouchables remaining untouchable. Michael Gray is forced to nofollow advertiser links.

Payola or Blogola, whatever you wish to call it still exists, and is practiced by Google.

Affect on Search Results?

When Matt Cutts defends Google’s actions because Google doesn’t need links, that isn’t quite the whole truth.

It is quite true that Google doesn’t need to rank for “search engine” in Google

Here in Poland, a search for “Android” which used to be a very generic term, the first 4 results point to sites about Google’s Android operating system.

But Google doesn’t rank for Mobile Phone, and even their partner, HTC who made both the G1 and G2 handsets only rank 3rd for smartphone, using US Geolocation and personalized search off (not that I search for this topic… ever), with Blackberry in 2nd.
Actually that was yesterday, looks like HTC now rank 2nd, and Blackberry has been pushed down the results.

Here are the current results for various terms:-

Smartphone

http://www.google.com/search?q=smartphone&pws=0&gl=US

Smartphones

http://www.google.com/search?q=smartphones&pws=0&gl=US

Smart Phone

http://www.google.com/search?q=smart+phone&pws=0&gl=US

Smart Phones

http://www.google.com/search?q=smart+phones&pws=0&gl=US

HTC Cloaking

Just try accessing this link which is the one that appears in search results – certainly from Poland I end up on different pages, based upon IP.

http://www.htc.com/ (I nofollowed the link – I am not going to link to a Blackhat site that is cloaking)

It is cloaking – users see different pages compared to search engines, though I am sure their SEO team hate the flash.

I see an English language snippet, and land on their Polish language site /pl/

The only way to see the root domain is in the Google cache.

With Google buying them links all over the blogosphere, they don’t need to worry, they don’t even need to buy PPC advertising, unlike Blackberry.

In a battle where HTC have only 180K links and Blackberry have 300K+, visitor data suggests Blackberry is still killing HTC, and other factors, the notion that Google’s partner doesn’t need more links is harder to excuse.

Btw Blackberry sell Smartphones -and don’t cloak their Smartphone website

Palm also sell Smartphones, but aren’t going to get links such as Smartphone or Smartphones unless they fix their funny redirects as well.

Then of course there is the IPhone – it would make a great TV gadget using Boxee, but all the plans in Poland offer at most 5GB of data – not interested. At least when I seach for Apple or Iphone in Google, I get given a link in the search results which is the page I end up on (in Polish). When I search with US geolocation, I get the US result for Iphone, which I click on and get the appropriate landing page.

p.s. Google no longer remembering my search preferences such as &pws=0&gl=US in both Firefox and Chrome is annoying me severely, and almost seems to be a deliberate change in implimentation.

p.p.s. This post does not contain any links for which I have received financial compensation. I haven’t received any compensation for this post from HTC (for the SEO review), Palm and Blackberry (for the nice rich anchor text) – if any of them decide to send me a free phone that won’t influence me to write about them again, and the test period due to infrequent usage might last a few years… but I am a “software developer” and “technology blogger”.
SEOmoz, Michael Gray and Fantomaster have linked to me and tweeted about my posts from time to time.

p.p.p.s Disclaimer:- I don’t class myself as an SEO consultant, this post is my personal opinion, and Google is the final decision maker over whether their commercial partner (HTC) is cloaking or not, and defines what is or isn’t a paid link. Maybe a expert on search engine cloaking could offer some advice.

Update

Just so we are totally clear over what is or isn’t allowed under the Google Webmaster guidelines, here is what Google stated in their official blogpost on the webmaster blog.

Geolocation: Serving targeted/different content to users based on their location. As a webmaster, you may be able to determine a user’s location from preferences you’ve stored in their cookie, information pertaining to their login, or their IP address. For example, if your site is about baseball, you may use geolocation techniques to highlight the Yankees to your users in New York.

The key is to treat Googlebot as you would a typical user from a similar location, IP range, etc. (i.e. don’t treat Googlebot as if it came from its own separate country—that’s cloaking).

Here is the video they also included:-

The only way I could get to the page Googlebot sees was looking at a cache on Google

UK Smartphone SERPS

Lots of SEOs seem to think brands have been pushed to the front of the SERPs, but that certainly doesn’t seem to be the case in the UK with the #1 manufacterer of Smartphones pushed to the 4th page of the SERPs potentially because they use “legitimate” SEO practices (there are a few things that need to be cleared up, hope their SEO team are working on it)

IP delivery can have significant benefits – if a UK user is forced to visit the UK site, even when clicking through from a US search result, the default link they will use will be to the page for the UK.

It is something that can be done without breaking Google’s webmaster guidelines.

Update 2

More from Michael on Google Being Biased, plus a followup from Lisa on Google Profiling SEOs As Criminals

Silly me, I should have also linked to Rae’s Blackberry News site

Seems both of them overlooked HTC’s cloaking

With SmartPhones now being looked on as a necessity (at least in the US among Tech bloggers) there is obviously a huge competitive market, thus any search spam should be heavily monitored by Google.