Please, don’t use Google hit counts to bolster arguments.

Don Kleinmutz, the President of INFORMS, wrote an interesting article on the failure of risk management in the recent issue of Analytics Magazine.  But to bolster a clearly justifiable argument, he wrote: “So are these pervasive and systematic failures of risk management something new or unusual?  No.  An Internet search of ‘risk management’ plus ‘failure’ produced 12 million hits on Yahoo! and 4 million hits on Google.”
First, it’s impossible to understand what the number of hits mean.  If the number of hits were 1/10th as large, would this be less compelling evidence?

More to the point, we have no clue what was on the pages or if it supports his conclusions, especially the last 99.99% of the pages. For example, I randomly selected the 5th page of results (which is in the top .002% of all hits), and looked at the first item.  It is entitled “Failure reduction in manufacturing systems through the risk management approach and the development of a reactive maintenance model.”
It’s tempting to use Google for justifications and it’s becoming widespread.  But it is also a bad idea. I can prove it.  I just typed in “Google” AND “bad idea” and got 1.84 million hits.

Advertisements

Tags: , ,

3 Responses to “Please, don’t use Google hit counts to bolster arguments.”

  1. Don Kleinmuntz Says:

    Jim,

    You make a fair point. By way of excuse, my use of hit counts was really just a rhetorical device to suggest that it is not hard to find examples of risk management failures. I agree with you, in hindsight, that this is a point that I could have made without hanging admittedly dubious numbers on it. As you say, my statement would not have changed if the numbers had been lower (or higher) by an order of magnitude — a sure sign of an uninformative number! Of course, if the search had come back with null results, it might have been more informative, but null searches involving common english language words and phrases do not seem to happen very often with current search engines.

    Was my search a useless excerise then? Although I didn’t mention it in my article, it did help me in a specific way. I have worked in risk analysis and risk management for some years now, so I have a tendency to see failures of risk management everywhere I look. My search was a useful starting point for testing the empirical validity of my list of subsequent examples of failures that have been attributed to poor risk management. For instance, I looked for the intersection of “risk management failure” and “Enron” to see whether I could find credible sources either in academic or business publications who had made this connection (a “no-brainer” in that specific case). I was not interested in the total number of hits, but rather, whether the connection had been made by sources I respected. I don’t recall the specifics, but in at least one case, I could not find credible sources and I did not use the example.

    Using the number of hits from Google as a justification is easy, but not something I am likely to do again. On the other hand, using Google (or other engines) to find sources to help verify an argument is quite useful, and something I am likely to do in the future.

    Best,
    Don

    PS: The article was actually a slightly reworked version of my President’s column in the February, 2009 issue of OR/MS Today. Several of my predecessors as INFORMS President used to wonder aloud as to whether anyone actually read these contributions. The nice thing about receiving comments, whether positive, negative, or somewhere in between, is that it provides proof that someone out there is reading what I have to say! Thanks.

  2. Jim Orlin Says:

    Don, thanks for being a good sport.

    I love using Google and find it really valuable.
    You could have made your point by referencing the first few hits that came up. Or possibly by doing a little more searching.

    The hit number for compound queries is notoriously difficult to interpret, and that was really my only point.

  3. Kaiser Says:

    Jim, I’m glad you wrote this little post. Every time I see this sort of argument, I cringe as well. As you said, one thing missing from all of these statistics (including everything that Google Analytics provides) is any notion of base rates, or any benchmark for comparison.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: