Web Performance Matters

Performance testing is the discipline concerned with determining and reporting the current performance of a software application under various parameters. But there comes a time after the tests are run when someone who's reviewing the results asks the deceptively simple question: So what, exactly, does all this mean? This point beyond performance testing is where the capabilities of the human brain come in handy.

With these words, Scott Barber introduced a series of articles on IBM's DeveloperWorks site about the human aspects of performance testing.

Beyond performance testing

In a 14-part series of articles published in 2004, Scott addresses ... what happens after initial test results are collected, the part it takes a human brain to accomplish, (and) what performance test results mean and what can be done to improve them.

In particular, he plans to ... focus on performance tuning, the subset of performance engineering that complements performance testing.

Illustrating the contrast between man and machine, he quotes Pablo Picasso -- Computers are useless; they can only give you answers -- and British journalist Jeremy Campbell:

Computers are good at swift, accurate computation and at storing great masses of information. The brain, on the other hand, is not as efficient a number cruncher and its memory is often highly fallible; a basic inexactness is built into its design. The brain's strong point is its flexibility. It is unsurpassed at making shrewd guesses and at grasping the total meaning of information presented to it.

-- Jeremy Campbell, Grammatical Man: Information, Entropy, Language, and Life (1982) Ch. 16

In the first article, he introduces the concept of performance engineering and provides an overview of the articles that follow:

About performance engineering

Performance engineering is the process by which software is tested and tuned with the intent of realizing the required performance. This process aims to optimize the most important application performance trait, user experience.

Historically, testing and tuning have been distinctly separate and often competing realms. In the last few years, however, several pockets of testers and developers have collaborated independently to create tuning teams. Because these teams have met with significant success, the concept of coupling performance testing with performance tuning has caught on, and now we call it performance engineering.

Let's begin by exploring this concept at a high level.

The performance testing part of performance engineering encompasses what's commonly referred to as load, spike, and stress testing, as well as validating system performance. You may also have heard other terms used to describe aspects of what I'm calling performance testing.

Regardless of the terms you may use to describe testing it, performance can be classified into three main categories:

• Speed -- Does the application respond quickly enough for the intended users?
• Scalability -- Will the application handle the expected user load and beyond?
• Stability -- Is the application stable under expected and unexpected user loads?

The engineering part comes in as soon as the first measurement is taken and you start trying to figure out how to get from that measurement to the desired performance. That's really what engineering is all about: solving a problem to achieve a desired and beneficial outcome. As a civil engineering student at Virginia Tech in the early 1990s, I was taught an approach to solving engineering problems that can be summarized as follows:

• Given -- What are the known quantities?
• Find -- What are the requirements for the desired item/outcome?
• Solution -- How can we build or acquire the desired item/outcome, meeting the requirements, within the parameters of the known quantities?

Or as my father used to say, "What have you got? What do you want? How do you get there?" In civil engineering, the implied problem was always "build or modify a structure to satisfy the given need." In performance engineering the implied problem may seem different, but it's fundamentally the same -- that is, "build or modify a computer system to satisfy the given performance need."

With that said, I suspect you would agree that performance is the trait of the system that we wish to engineer, thus performance engineering.

-- Scott Barber, Beyond performance testing: Part 1, Introduction (2004)

Here are Scott's descriptions of the content of the next 13 parts, organized under three major headings:

Performance engineering housekeeping

Parts 2, 3, and 4 start us out with what I think of as performance engineering housekeeping topics. These three articles aren't particularly technical but will give us the common footing we need for the remaining articles.

• Part 2 A performance engineering strategy -- The performance engineering methodology that serves as a basis for applying the concepts in later articles is described here. This article also outlines a performance engineering strategy document (equivalent to a functional test plan) that can be used to help organize and document performance engineering activities, plus a performance engineering results document.
• Part 3: How fast is fast enough? -- One of the biggest challenges we face in performance engineering is collecting performance-related requirements. This topic was touched on in the "User experience, not metrics" series and is discussed here in more detail.
• Part 4: Accounting for user abandonment -- If an application is too slow, users will eventually abandon it. This must be accounted for in both modeling and analysis, because not accounting for abandonment can drastically change your results and tuning strategies.

Detecting and tuning bottlenecks

Parts 5 through 10 are technical articles focused on detecting, exploiting, and preparing to tune bottlenecks. These articles follow the steps of the approach we discuss in Part 2 and continually relate back to the performance requirements we discuss in Part 3.

• Part 5: Determining the root cause of script failures -- The obvious symptom is very rarely the actual cause of a script failure. It's imperative to determine if the failure is due to the script or the application and be able to quantify the root cause, not just the symptoms.
• Part 6: Interpreting scatter charts -- Scatter charts are the most powerful evaluation tool at a performance engineer's disposal and must be used wisely to pinpoint performance issues. This article expands on discussions of the scatter chart in the "User experience, not metrics" series.
• Part 7: Identifying the critical bottleneck -- One part of the system is always slowest, and until you remedy that bottleneck, no other tuning will actually improve the performance of the application along that usage path. Before you tune it, you must first conclusively identify it.
• Part 8: Modifying Tests to Focus on Failure/Bottleneck Resolution -- Once a failure or bottleneck is found from a functional perspective, resolution can be reached more quickly if you modify your existing tests to eliminate distraction from ancillary issues.
• Part 9: Pinpointing the architectural tier of the failure/bottleneck -- Just because you've identified a bottleneck or failure from a functional perspective and can reproduce it, doesn't mean you know where it is. Pinpointing exactly where the bottleneck is can be an art all its own.
• Part 10: Creating a test to exploit the failure/bottleneck -- Now that you know what the bottleneck is functionally and where the bottleneck is architecturally, you will likely need to create a test to exploit the failure/bottleneck in order to help the developer/architect with tuning. This test needn't bear any resemblance to real user activity but rather needs to exploit the issue, and that issue alone. In fact, these scripts often don't even interact with the system in ways that users would and may include direct interaction with back-end tiers.

Advanced topics

Parts 11 through 14 are "clean-up" topics. These are areas that are referred to in one or more of the previous articles but not done justice due to topic and length constraints. These articles expand on topics that are applicable throughout the engineering process.

• Part 11: Collaborative tuning -- Once you can exploit the failure or bottleneck, you need to know how to work with the rest of the team to resolve it.
• Part 12: Testing and tuning on common tiers -- Web, application, and database servers are common tiers related to testing and tuning. Here we discuss specific issues, methods, and resolutions for these areas.
• Part 13: Testing and tuning load balancers and networks -- These components require a significantly different approach from that called for by common tiers. This article discusses issues specific to these areas and approaches for handling them.
• Part 14: Testing and tuning security -- Security is the most performance-intensive activity of an application. Tuning security issues often involves taking a risk-based approach to balancing necessary security with desired performance. [I couldn't find the last part on the IBM site, but the linked version seems to be Scott's original text anyway.]

If you're at all interested in performance testing, I thoroughly recommend these well-written articles.

This post is one of a series on fundamental truths about performance.
Tags: performance wisdom, performance engineering, Scott Barber, IBM, testing, software testing