Web Performance Matters

Previous posts in this series described the influence of Moore's Law and Wirth's Law on application performance, how to balance hardware capacity and software demand with a systematic approach to performance tuning, and the five fundamental elements of computer system performance.

Today's post is the first of several that will review insights on the central importance of measurements. Previously I've described how I view measurements as the foundation of all performance management -- see, for example, The ABC's of Measurement Data and my review of Practical Service Level Management.

Today's quote by Lord Kelvin sums up my point of view. Unless you can measure something, your attempts at managing it, and maintaining or improving its performance, will be unscientific at best.

In fact, because their work depends on their ability to collect and analyze data, I think most performance analysts would agrees with the popular saying that "you can't manage what you can't [don't] measure".

But just who said that?

Interestingly, despite my strenuous efforts to track down the origin of that widely quoted saying, its source remains obscure. As I wrote in a previous post:

Perhaps because of my training in statistics, I had always assumed it was just a simplification of a famous statement by Lord Kelvin, often cited by statisticians:

But as I discovered after much research online:

Unfortunately, there does not seem to be a lot of support for my assumption. Instead, my Web searches have revealed that the statement 'you can't manage what you can't measure' is most often attributed either to the pioneer of quality control, W. Edwards Deming, or to 'the father of modern management', Peter Drucker. Moreover, neither of these popular attributions seems to be correct.

See that post -- Deep Thoughts on Management -- for a few more details.

Because I wrote that post a year ago I repeated my search today, but reached the same conclusion. Although Drucker is sometimes credited, there is no credible evidence to support his authorship. The closest thing I have seen to an actual citation is in a 2004 paper on Web Log Analysis: A Study of Instructor Evaluations Done Online [pdf] by Kenneth J. Klassen and Wayne Smith. Their paper's introduction and references contain the following citation:

While the sage advice “you can’t manage what you can’t measure” (Drucker, 1993) is still very much true, in an e-Business environment it is also true that “you can’t measure what you can’t monitor”.

Drucker, P. (1993). The Practice of Management (Reissue edition). New York, NY: Harper Business.

On the face of it, this looked like a breakthrough. So I dug a little deeper ...

Peter Drucker on measurements

While I do not own a copy of that book, I do own The Essential Drucker [Amazon], a collection of excerpts drawn from ten of his books. And Chapter 8 is based entirely on The Practice of Management. In it he writes:

To be able to control his own performance, a manager needs to know more than what his goals are. He must be able to measure his performance and results against the goal. It should indeed be an invariable practice to supply managers with clear and common measurements in all key areas of a business.

Those measurements need not be rigidly quantitative; nor need they be exact. But they have to be clear, simple, and rational. They have to be relevant and direct attention and efforts where they should go. They have to be reliable—at least to the point where their margin of error is acknowledged and understood. And they have to be, so to speak, self-announcing, understandable without complicated interpretation or philosophical discussion.

Each manager should have the information he needs to measure his own performance and should receive it soon enough to make any changes necessary for the desired results. And this information should go to the manager himself, and not to his superior. It should be the means of self-control, not a tool of control from above.

This needs particular stress today, when our ability to obtain such information is growing rapidly as a result of technological progress in information gathering, analysis, and synthesis. Up till now more information on important facts was either not obtainable at all, or could be assembled only so late as to be of little but historical interest. This former inability to produce measuring information was not an unmixed curse. For while it made effective self-control difficult, it also made difficult effective control of a manager from above; in the absence of information with which to control him, the manager had to be allowed to work as he saw fit.

Our new ability to produce measuring information will make possible effective self-control; and if so used, it will lead to a tremendous advance in the effectiveness and performance of management. But if this new ability is abused in order to impose control on managers from above, the new technology will inflict incalculable harm by demoralizing management, and by seriously lowering the effectiveness of managers.

-- Peter Drucker, The Essential Drucker, Harper Business (2001) pp 120-121.

So Drucker clearly believes that measurements are important, but also accepts that the right measurements may not always be available when they are needed. Granted, this is not the entire book. Granted, he wrote this in 1954, and the use of computers and metrics in management has evolved a lot in the last 50 years. But if this represents what he believed in 1954, I seriously doubt that the same book, or even a revised edition published in 1993, would have also included (maybe in another section) the black-and-white statement that you can't manage what you can't (or don't) measure.

If he had made that exact statement, surely it would have appeared somewhere among the paragraphs quoted above? And since the saying is so well known these days, wouldn't it have been included among the many extracts selected for inclusion in "The Essential Drucker"? The fact that it was not -- together with its absence from other online lists of famous Drucker quotations like Wikiquote and BrainyQuote -- supports my conclusion that he did not actually say it.

Wisdom or fallacy?

Despite the widespread acceptance of this saying by almost everyone who repeats it, not everyone agrees that it states a universal truth:

• Drucker's comments signal his concerns about possible misuses of measurements
• Others take Deming's view that managers must manage what they can't measure

Robert L. Glass goes so far as to cite the popular saying as one of the "Fallacies of Software Engineering Management", in the section of his book about such fallacies. According to Glass:

The saying "you can't manage what you can't measure" appears most frequently in books and articles on software management, software risk, and (especially) software metrics. An interesting thing happened when I set out to track down where the saying originally came from. Several metrics experts said that it came from Controlling Software Projects (DeMarco 1998), and so I got in touch with Tom DeMarco himself. "Yes," said DeMarco, "it's the opening sentence in my book, Controlling Software Projects. But," he went on to say, "the saying is actually 'you can't control what you can't measure.'" Thus the fallacy version of the saying is actually a corruption of what DeMarco really said!

-- Robert Glass, Facts and Fallacies of Software Engineering, Addison Wesley 2003

Frankly, I am highly skeptical of this explanation, which strikes me as reflecting a particularly myopic worldview. I don't doubt that Tom Demarco's 1998 book opened with the statement "you can't control what you can't measure." And maybe the "several metrics experts" who Glass consulted first encountered it there. But I also believe that this saying has been around for longer than 9 years. So it seems to me far more likely that DeMarco adapted a well-known adage than that the rest of the world picked it up from his 1998 book about software project management.

Update #1: I see that DeMarco's Controlling Software Projects [Amazon] was first published in 1986, not in 1998 as Glass states. A customer review confirms that it opens with "you can't control what you can't measure." This makes it more possible that DeMarco was indeed the original source.

The mystery deepens!

Update #2: The introduction to the Wikipedia article on Software metric states:

Tom DeMarco stated, “You can’t control what you can't measure” in DeMarco, T. (1982) Controlling Software Projects: Management, Measurement & Estimation, Yourdon Press, New York, USA, p3.

It seems likely that this citation refers to an even earlier edition of DeMarco's book, adding to the possibility that he really was the original author of the saying. But if so, who then altered it from "you can't control" to "you can't manage" -- the form that everyone now quotes?

Request: If anyone has any information about this question, please post it in a comment below, or in the Discussion Forum, where I have created a post on this topic. Even if the saying was based on Drucker's ideas -- or DeMarco's book -- someone must have coined the particular version everyone knows today. I'd really like to resolve this mystery once and for all, and give the credit where it's due!

The bottom line ...

I still believe that, where computer system and application performance is concerned, it is very difficult to manage anything you don't measure. I still agree with Kelvin that ... when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.

But the line of thinking advanced by Deming and Glass has also led me to search for some slightly narrower, perhaps more focused, statements about the role of measurements. Those will be the the subject of further posts in this series.

This post is the fourth in a new series on fundamental truths about performance.
Tags: performance wisdom, performance, Lord Kelvin, Kelvin's dictum, Drucker, Deming, software performance, measurement, management