Web Performance Matters

Posts on The Importance of Measurements and Controlling Software Projects have reviewed the origin of the saying that "you can't manage what you can't (or don't) measure". Today I look more closely at its meaning and validity -- how true is it?

One apparent contradiction is that this much quoted fact of management is also widely viewed as a fallacy -- or at least, as an over-exaggerated claim -- especially by people in the software engineering profession, which seems (in the person of Tom DeMarco) to have coined the saying in the first place. That contradiction was highlighted in a 2003 book by Robert L. Glass, Facts and Fallacies of Software Engineering [Amazon].

In Chapter 5, reproduced as Fallacies of Software Engineering Management at Informit.com, Glass cites the popular saying as the first of five management fallacies. According to Glass:

The problem with the saying "you can't manage what you can't measure"—what makes it a fallacy—is that we manage things we can't measure all the time. We manage cancer research. We manage software design. We manage all manner of things that are deeply intellectual, even creative, without any idea of what numbers we ought to have to guide us. Good knowledge worker managers tend to measure qualitatively, not quantitatively.

-- Robert Glass, Addison Wesley (2003)

But we really do need measurements

Nonetheless, Glass actually agrees with the intent of the saying. His only real objection is to its form, which he cautions, "should not cause us to reject the underlying truth of the message it brings". He goes on to say:

Managing in the presence of data is far better and easier than managing in its absence. In fact, it is the nature of managers—and human beings in general—to use numbers to help us understand things. We love batting and fielding and earned run averages. We love basket and rebound and assist counts and invent terms like triple double to accommodate combinations of them. We even invent data to cover subjects when there is no natural data, such as ice skating and diving (where judges assign scores to performances). ... The fact is that measurement is vitally important to software management, and the fallacy lies in the somewhat-cutesy saying we use to try to capture that.

-- Robert Glass, Addison Wesley (2003) [emphasis added]

In his own reviews of the book [bergware.net, biblio.com], Glass explains that "I suppose I have to admit that the things I call fallacies are things that others might call facts. But part of your fun in reading this book should be forming your own opinion on the things I call facts and the things I call fallacies."

Despite his ambivalent stance on the issue of fact vs. fallacy, Glass has evidently struck a nerve among software engineers in this instance. A typical example of the debate occurred in Catenary, the blog of graduate computer science student Jorge Aranda, who wrote:

I recently came across Tom DeMarco's "Controlling Software Projects" for a second time, and I remembered my problem with it immediately: The very first line in the book states that "You can't control what you can't measure", and the rest of the text builds upon that phrase to argue that we need metrics to rein the chaos of software development.

But the book falls apart because the statement is wrong, and you and me (sic) are living proof of that. We control things we don't measure, all the time. You are able to control your body (your breathing, your hunger, your thoughts) without measurements, and you are an expert at it. In some cases measurements can help—for example, for controlling your weight, or your cholesterol. In many others, however, measurements would not even make sense (do you need to measure the length and number of your hairs to know when to get a haircut?)

What bothers me is that the phrase has picked up among software project managers, and is often used to justify absurd metrics and policies. Relying on measurements such as lines of code for programmer efficiency, or number of errors found for software quality, are excellent ways to lose control over a project. They are deficient and misleading proxies for the real constructs that one wants to study.

Must measurements be quantitative?

But in the end, Aranda also has to admit that the traditional saying does have some merit:

There is a kernel of truth to the phrase, however. The unknown is, by definition, uncontrollable, and sometimes numbers help us increase our knowledge. But I would much rather read a software development book that relaxes the statement to something like "You can't control the unfamiliar", and builds a thesis from there. It doesn't have the same punch, but it has the benefit of being true.

He concludes by citing Robert Glass and his view that the saying is a fallacy. Commenter and fellow blogger Manuel Berlanga responds, supporting Glass's viewpoint that "good knowledge worker managers tend to measure qualitatively, not quantitatively". He suggests that the saying can be interpreted to include qualitative measurement:

The idea that you need to measure everything to control it makes sense when you include non-quantitative measurements. You don’t measure your hair but you look at a mirror and compare its length to you usual standard. That is a way of measure that relies on qualitative standards. You can measure the amount of food you have by the sensation of fullness and, with hope, happiness of your belly. Not all measures have to be in ones and zeroes, but DeMarco’s work misses this fact and (is excessive when it advocates) measuring the efficiency of a worker by the lines of code he writes.

Jorge responds, agreeing that "if we expand the definition of measurement to include qualitative assessments, then I have no problem with the phrase. But the standard meaning of the word calls for numbers, and that’s the way the phrase is interpreted most of the time!" This is a very reasonable objection, as is evident from the Wikipedia article on "Measurement":

In addition to the standard use of the word measurement, the word is often loosely used to mean any number assigned to a physical object. Thus the wind chill factor, which has no scientific validity, or IQ, which is just a count of correct answers on a test, are loosely called measurements. Measurements carried out by surveys or questionnaires are, strictly speaking, counts rather than measurements ...

-- Wikipedia article on Measurement [*]

[*] Update: Although the quoted text was removed by an edit on June 23, 2007, the present article retains the notion that measurement is not merely the assignment of a value. -- July 11, 2007.

A matter of semantics

So what do you really mean when you say "you can't measure"?

I believe this semantic issue underlies all debate and disagreement about the truth or falsity of the saying "you can't manage what you can't measure." Because everyone who either cites it, or objects to it, does so in the context of two (often unstated) assumptions about:

1. The subject matter or domain to be managed
2. The nature of any possible measurements

For example, a book review by Charles Ashbacher states:

His first fallacy, “You can’t manage what you can’t measure”, is clearly very controversial and one that I am in complete agreement with. A great deal of money has been spent on tools to measure the progress of a software project, which means that many people, sellers and users alike, have a large stake in believing that measurement is a complete solution. However, there is strong evidence that every metric used to measure software development fails in many circumstances and there is overwhelming evidence that many groups simply ignore the measurements and suffer no ill effects.

-- Journal of Object Technology, vol. 2, no. 1, Jan-Feb 2003, pages 119-120

Here the context is the use of software metrics, but Ashbacher bases his conclusion on the use of "tools to measure the progress of a software project". Clearly his choice of context has influenced his conclusion.

But what if the measurements in question were not generated by a "tool", but were the percentage of successful functional tests conducted by an experienced QA team working independently of the engineers? Is the saying still a fallacy, or would Ashbacher admit that it would actually be possible for such measurements to at least help a manager assess the progress of a software project? In fact, wouldn't having those counts be a lot better than not having them, and risking the possibility that the engineers were supplying overly-optimistic estimates of their progress? My own experience participating in, and managing, software engineering projects tells me it would be.

And I have not even touched on all the technical management activities that absolutely require measurements. Service level management and performance, for example!

The bottom line

So my own conclusion about this "controversy" is summed up by Catenary contributor Mike, who writes:

I prefer what Tom Gilb said, “Anything you need to quantify can be measured in some way that is superior to not measuring it at all".

I believe this clever statement by Tom Gilb, author of the classic 1988 textbook Principles of Software Engineering Management [Amazon] provides the perfect answer to all debate about this issue.

It cuts through any arguments from the outset, limiting the scope of its conclusion to anything you need to quantify. Having homed in on something that needs to be measured, it makes what I believe to be a very safe claim: no matter what the domain of interest, if you need to measure something, there will always be some way to measure it that will tell you more than you would know if you were not to measure it at all.

That's why Tom Gilb's observation is today's featured Performance Wisdom. It gets across the value of measurements while avoiding all the semantic pitfalls associated with the more well-known (and more contentious) alternatives.

This post appears in both the management wisdom and performance wisdom series.
Tags: performance wisdom, performance, management wisdom, management, Tom Gilb, software engineering, measurement