Google Killing Off Automated Queries – num=100 R.I.P ?

Just a few days ago I reported that I was included in a random test where I couldn’t create queries for less than 20 queries.

Now we have had Google Instant launch and people have been playing with it for 24 hrs or so.

One of the things I have spotted is that currently, it is really hard to get more than 10 queries per search for normal users.

no num100 automated queries

I have tried simple queries like this

http://www.google.com/search?q=andy+beard&num=100

I have tried by seting my search preferences to 100 results

I have tried with Google Instant switched on and off

I have tried both logged into Google, and logged out

Currently the only combination that allows me more than 10 search results of all those possible factors is logged out, with Google instant swiched off.

That could easily mean that Google will leave a window for scrapers to get 100 results, I could be part of another test group, or this could be a bug in the aftermath of the test group I was included in.

It is not an option I use very frequently, but it is something used heavily for rank checking tools.

Technically, automated queries against Google’s search index that don’t utilize Google’s Search API are a violation of Google’s terms of service and webmaster guidelines.
Some might claim that Google scrape their content without permission – the difference is that Google adhere to the REP (Robots Exclusion Protocol) – you can block them with a robots.txt file in the root directory of your domain.

However many webmasters want to monitor their search rankings with tools (Google provide some data in webmaster tools of questionable accuracy) – the only viable way to do it is with automated tools.

If you happen to work for an INC500 with rules about not breaking Google terms of service, but you have to provide reporting on ranking… you end up doing the queries manually and recording them in a spreadsheet.

I am hoping this is just a bug that will eventually be fixed… what do you think?

p.s. still preparing a longer Google Instant opinion piece.. watch this space

Update

Firstly David Iwanow from The Lost Agency identified the parameter that can be used to persuade Google to show 100 results.

&as_qdr=all

Yes it is a date range parameter – i.e. a work-around that currently works

And the guys from Online Sales, champions of SEO Bookmarklets have come up with one to show 100 search results with personal off

Liked this post? Follow this blog to get more. Follow

Comments

  1. says

    Hey Andy,

    It does still work you just have to tweak the result that includes “&as_qdr=all”

    The following link does a sample query showing 100 results…
    NYG Gin

    • says

      David killer solution

      I have some guy on the Google search help group effectively saying I am an idiot and this post is inaccurate

      Blogging about a problem (after sufficient reasearch so you don’t look like a total idiot) and finding a solution from a smart reader is a far better solution.

      • says

        Ah damn that’s fairly harsh i mean its technically correct google killed the old method, but i’m going to leave the research to someone else to figure out if the new way I discovered was done because it was better technique for Google’s own data or might have a unknown impact on how results are presented in the top 100.

        I think he could have done the same thing I did and look at the issue and tried to see if they could have tried to think about the issue.

        really not liking negative karma in the Google groups, as it unless its a specific statement such as getting web traffic from Bing helps your Google rankings…

        Glad you like the solution.

        • says

          I don’t go to the Google groups very often, but in this case Barry had started a thread about the 20/30 results we were seeing, he was seeing responses from the usual “you are doing it wrong” crowd, and I chimed in just to give support.

          I just decided to tag on the end of it after Instant was launched that there are other issues.

          Groups are great in that you can get direct feedback from Googlers, but their answers have to be suitable for general consumption, thus I have always found them to be too strict to the official lines and anything in the grey area is regarded a bit taboo.
          Grey as in not clearly defined, no grey as in greyhat… though that might be just doing stuff that Google hasn’t got rules for yet.

  2. says

    Thanks Andy. I’m not too sure what’s going on with Google Instant yet, but will keep a close eye on what you report and a look out on other sources too.

  3. says

    I wonder how long it will take for common SEO tools like SEObooks Rank Checker plugin to be adjusted to accommodate these changes. Currently, through tools like these, all results come back as ZERO and yet in Yahoo and Bing they are still there.

    As an SEO our ability to provide uninterrupted updates about rankings is imperative to our customers. When large shifts in search take place you have SEO’s across the world who could simply revolt and push their customers to begin using Bing as an alternative to Google.

  4. says

    I don’t know about you, but I realized that &num=100 changed the rank order of some keywords, so I gave up on trying to use it for automation. I just retrieve SERP pages one-by-one just like a real user.

    • says

      That is working as expected.

      One of the best examples is with double listings which get rolled up if someone had multiple results for the same query that qualify to be in those first 100 results.

      What I would typically do (or suggest others do who are good at programming) is get the results for the first 10, and then use 11 to 110 to look for opportunity for double listings, universal search etc.

  5. says

    Hi Andy,

    Interestingly, when I use Advance Search and set it to 100 results, I still get the old search? url rather than the new Ajax one (and it includes the &as_qdr=all). I wonder why they’ve dropped it from the Ajax URL? I can’t see any reason to leave it out.

    Anyway, Instant’s given us all a lot to think about. Oh, great, I just realised that it’s broken my Google Numbered Search Results Greasemonkey script. I suddenly care about this a whole heap more than I did :(

  6. says

    One thing we noticed while investigating Google Instant was that they write preference in the cookie file. In Chrome we browsed to: Options > Under The Hood > Content Settings > Show Cookies and Other Site Data… > Search: google.com > Expand Cookies > Open “PREF” entry.

    Google Instant enabled: SG=2
    Google Instant disabled: SG=1

    This could be useful when retrieving Google data through a “browser” signature with a cookie.

  7. Wards says

    It’s working for me with Instant turned off today (23/10/2010)
    DOes anyone know how to show over 100 results? Such as 300+????

Trackbacks