What I am about to reveal in some ways is akin to publishing 0-day security stuff, and some SEOs will look on this as revealing things that they have been happy to exploit for months – thus some of this is fairly widely known, but maybe you don’t.

I will also include some thoughts on “the reset vector” and “evaporating PageRank” in later parts (my views radically differ to those widely publicised – even at recent conferences)

Lots of this article series is not based upon verifiable facts or even poor attempts to construct tests that would be ripped apart anyway. I am just throwing my thoughts onto the interwebs.

I have decided not to do this as a video – some of my loyal readers have accessibility problems with videos and I don’t want to exclude them.

This is a little technical and some aspects would be highly dependent on how Google treat internal linking within Twitter. It is hard to exclude external factors as there are very few ways to create a controlled experiment and there are so many possible ways to filter trust based on influence within the Twitter ecosystem and the whole social graph.

Robots.txt

#Google Search Engine Robot
User-agent: Googlebot
# Crawl-delay: 10 -- Googlebot ignores crawl-delay ftl
Disallow: /*?
Disallow: /*/with_friends

This code prevents Googlebot crawling your historical tweets
Individual tweets if they get some juice from elsewhere can be indexed, but twitter aggregators tend to nofollow links to the content they repurpose.

You would think as Google crawl robots.txt on every visit, they would always have a copy of the robots.txt available… well unless you used x-robots in some way to nocache it.
You would also expect it to be reasonably up to date.

However here is a link to their current robots.txt as visible to humans, and it matches the cached version, and the version I have on this page above.

Interesting Factoid

Twitter doesn’t curently rank in the top100 Google results for Microblogging
http://www.google.com/search?q=microblogging&pws=0&hl=en&gl=US&num=100

So What Does Twitter Want To Rank For?

Quite probably Twitter’s only goal is to rank for your name or your brand.

In fact if you don’t have a Twitter account for your brand that is actively used (it is fairly safe to squat), it is highly likely that you will never have Twitter appear with realtime results for brand searches in the main SERP unless you are a trending topic.

Once you invite social media into your house, it is hard to get rid of.
(it isn’t as bad/good as it was a few months ago)

I noted to some friends when Ikea was having some social media unrest, and had just announced something so lots of press attention, they still didn’t have a scrolling realtime search box appearing in their brand search engine results, whilst it might only take a mention of a popular Twitter user’s name a few times, maybe a few retweets of a blog post for it to normally appear.

I assume Ikea own their trademark account or someone is being very careful squatting it (maybe even Twitter management) – it has PageRank – it has some followers – but it isn’t active.

The Twitter account currently ranks ~181 in the US search results on Google, just behind quite a recent article on Huffington Post – oh and I just linked to it.. oops.

Considering the strength of the brand, that is a pretty strong ranking without any further support such as the link I gave it.

Real Time SERPs

There have been some changes to realtime SERPs recently which Dave covered.

Here you can see that the people attending the Mozinar went to bed fairly early (must believe in the witching hour on the NW coast of the US, or the bar closed)

However unlike a few months ago it is very hard to get realtime results to appear in the main search results pages.

Stay tuned for part 2 – we are going to look at the internal linking structure, and how to leverage it to maximum effect either for your own brand, or someone else’s.

As I have mentioned before, I could understand and even encourage Google to somehow decide to pull in my most recent blog posts as part of their real time search. Most people searching for me it is most likely a regular reader doing a navigational query looking for my latest blog posts, not my latest tweets.

I see this with both a UK and US search, personalization switched off in the query, and logged out

http://www.google.com/search?q=andy+beard&pws=0&gl=US
http://www.google.co.uk/search?q=andy+beard&pws=0&gl=UK

This is an “Anytime” search – no need to select “latest”

Update – Chris Brogan Too

I wonder where Chris would prefer the traffic to go, blog or Twitter?

I can’t track search traffic to Twitter