Web Performance Matters

Recent posts have discussed some insightful statements about the importance of measurements by Lord Kelvin, Grace Hopper, Tom DeMarco, and Tom Gilb.

In the last of these, I concluded that Gilb's observation (Anything you need to quantify can be measured in some way that is superior to not measuring it at all) gets across the value of measurements without making any claims that are too far-reaching or contentious.

A follow-up comment and the ensuing conversation with Ben Simo -- author of Quality Frog, a blog about software testing and software quality -- reminded me of this post, which I'd been meaning to complete and publish for a while. I'll explain the reasons for the delay below.

But first let me quote the portion of Ben's comments that triggered this post. I won't repeat the whole conversation, but we were discussing whether metrics could be dangerous. The key part was this:

Metrics are not dangerous on their own. It is misuse of metrics that can be dangerous. Using the wrong metrics appears to be very common. Even reporting the right numbers without the accompanying story to put them in context can be dangerous.

-- Ben Simo

This reminded me of what I had written ten years ago in my book, immediately after introducing the five elements of performance on the third page of the chapter on Performance Fundamentals:

The Law of Measurements

The result of any measurement will depend upon what is measured, how the measurement is done, and how the results are computed.

We call this the law of measurements. This self-explanatory law may seem rather silly at first. But it is amazing how often and easily we can be misled by software measurements whose authors, sometimes deliberately, omit from their report of the results crucial details about the measured hardware and software environment, the exact workload used, the measurement methodology and its accuracy, or the methods used to compute the final results from the observed data.

As Benjamin Disraeli said, long before the modern computing era ever began, "There are three kinds of lies: lies, damned lies, and statistics". Notice the sequence! Lies disguised as detailed measurements can be especially beguiling.

-- Chris Loosley, Guideline 3.1, High-Performance Client/Server, Wiley (1998).

The importance of context

When I wrote this, I was thinking about misleading uses of data. Ben's comment clearly agrees with this, but also introduces an important fourth clause which could be appended to my original "law of measurements," namely:

The result of any measurement will also depend on how the results are interpreted.

And without a clearly stated context (or metadata) even the right data, measured in the right way, can still be misinterpreted. Perhaps accidentally, perhaps deliberately -- it may not matter, since the consequences may be the same.

In fact, Ben recently wrote a nice post in which he enumerated the kinds of metadata that should be collected while testing software, to enable proper analysis of the results.

Another authorship question

So it turns out that I'm now glad I didn't publish this post previously, because Ben's insight has improved the result. But what was my original reason for putting it on the shelf?

The problem was in the title. Back in March, when I started writing a series of posts about measurement, I was going to cite Benjamin Disraeli, whom I had always accepted as the author of the phrase, Lies, Damned Lies, and Statistics. I first saw this attribution in the second volume of Kendall and Stewart's three-volume textbook, The Advanced Theory of Statistics during the 1960's, and it was later confirmed by my Concise Oxford Dictionary of Quotations (1981 ed.).

Some people mistakenly attribute the saying to Mark Twain, but Twain himself ascribed the saying to Disraeli in Chapters from My Autobiography -- see the Wikipedia article.

But while checking my sources online, through Wikipedia I found an interesting page by Peter M. Lee, a statistics professor at the University of York in the UK. After some research, his tentative conclusion was that the phrase had originated with Lord Courtney, President of the Royal Statistical Society in 1897-9.

As evidence, he cited an 1895 speech in which Courtney said, "... although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics' ..."

Digging deeper ...

So I downloaded the pdf of the original speech by Courtney. But after reading it (see page 4 for the key quote), I was not entirely convinced. Courtney, I thought, might have been referring to an already familiar quote by an actual "Wise Statesman," as opposed to an anecdotal one. And if so, wasn't it quite possible that the wise statesman Courtney was referring to was in fact Disraeli, who before his death in 1881 could certainly have fit that description?

Realizing that Peter Lee's conclusions may be based on more information, I decided to do a bit more digging myself. Searching Google for "Wise Statesman" revealed that I am probably about 10,000th person to tread this well-worn online path, a few hundred of whom have also written something about it! But I did find some solid evidence (here and here) that linguist Stephen Goranson at Duke has been doing some pretty serious research work on this question.

I forwarded these references to Peter Lee, who updated his page. He now notes that a January 1892 quote by Sir Robert Giffen (1837-1910) seems now to be the earliest recorded use of the saying. But as he also said in an email, ... he still leaves it in doubt who originally coined the phrase. It seems a never-ending search.

It's all about the journey

This is what happens sometimes, when you start looking for the truth online -- you are drawn into an endless search, and wind up somewhere far away from what you thought was your original destination. A bit like life, really. And ultimately, the pleasure is in the journey, not the final destination.

Now, what was I talking about when I started this post ...?

This post is one of a series on fundamental truths about performance.
Tags: performance wisdom, performance, Ben Simo, Quality Frog, measurement,
law of measurements, Disraeli, Mark Twain, Lord Courtney, Robert Giffen, Stephen Goranson,
lies, damned lies, statistics.