Web Performance Matters

jQuery Library Performance Alert

Chris Loosley — Fri, 21 Aug 2009 22:51:53 +0000

jQuery libraries moved to Google AJAX Libraries API

Recently I've been doing a lot Web design and development work, using the Squarespace platform, a "A fully hosted, completely managed environment for creating and maintaining a website, blog or portfolio." I like Squarespace because it is xhtml/CSS based and lets me focus on a site's content and appearance. I get great performance and never have to deal with installing and managing any Web server software. Normally ...

Last night was the exception. I was working on some site updates, and every time I refreshed a page, there was an interminable delay while the page loaded. Coincidentally, I've been experiencing intermittent short outages in my AT&T U-verse service lately, so I put this down to another AT&T problem. But it was already after midnight anyway, so after trying a few things, I finally gave up and went to bed.

This morning, all was explained, thanks to a post about a recent jQuery change on the Squarespace Community Forum, by Stuart Hobday, a Web designer in the UK who (naturally) also uses Squarespace.

jQuery is a lightweight JavaScript library that many people find extremely useful for coding site UI behaviors, because its syntax lets you focus on the relationships between JavaScript, HTML, and CSS. That is why it is now very popular among Web developers who rely on JavaScript or Ajax frameworks to animate their sites.

When a Web page contains JavaScript code that uses a library, the library code must first be downloaded to the client. jQuery has always offered Web developers two ways to do this: download the code and upload it (along with the rest of your content) from your own server, or let the page download it directly from a jQuery server located at code.jquery.com.

The first approach is ideal once your site is in production and everything is working, but while you're in development, it's convenient to be able to grab the latest library code and be sure that you'll automatically get the benefit of any recent bug fixes or enhancements. So I think a lot of developers take that approach. And I imagine that there's quite a lot of jQuery code out there, in every day use, and still relying on a download from code.jquery.com.

And here's where the performance problem arises. Yesterday, that library was relocated to ajax.googleapis.com:

code.jquery.com Redirected to Google Ajax APIs
Posted August 20th, 2009 by Mike Hostetler
Starting at 10PM MT on August 20th, code.jquery.com will start redirecting (301) to ajax.googleapis.com [http://code.google.com/apis/ajaxlibs/documentation/index.html#jquery].

Immediate Impact:

None

Redirection will occur using 301 “Permanent Moved”

Packed version will be replaced with minified version

Long Term:

Migrate any sites using code.jquery.com to Google’s AJAX Libraries API

Full documentation of Google’s Ajax API are available at http://code.google.com/apis/ajaxlibs/documentation/index.html#jquery. For your convenience here are the old URLs on code.jquery.com and their new Google Ajax API counterpart:

jquery-latest.js

http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.js

jquery-latest.pack.js

http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js

jquery-latest.min.js

http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js

jquery.js

http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.js

jquery-1.3.2.min.js

http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js

...

--for the full list, see the jQuery blog

Although the old downloads still work, it's been taking code.jquery.com a long time to issue those 301 redirects to the google library. In the meantime, your browser will sit there (probably with a message at the bottom saying "waiting for code.jquery.com" or something similar). If you see that, look in the source code, usually in the section, for a jquery download that may be causing the problem. If you control the code, fix it. Otherwise call the tech support for the site and tell them what to do.

Tags: jQuery, performance, ajax, Google, download time, Performance Matters, Web performance

Where Performance Meets Availability

Chris Loosley — Mon, 09 Feb 2009 08:01:00 +0000

Response Time Standards for Web Sites and Web Applications

Earlier posts about Acceptable Response Times have discussed how a Web site or application's responsiveness can Delight, Satisfy, or Frustrate customers.

Availability, on the other hand, is a measure of a system's stability. It is not a performance metric, it is a software (or hardware) quality metric.

So, technically speaking, performance and availability are orthogonal issues. You would not call a London taxicab a high performance vehicle just because it runs for 300,000 miles without ever breaking down. Conversely, for broken software, or a car that will not start, performance measures are irrelevant.

Practically speaking, however, availability and responsiveness are interconnected concepts. How slow does a user interface have to be before the application becomes unusable, and might as well be completely unavailable? Slowness matters; how long will you wait before you click away? Maybe you'll come back later if you really need to see that page. If not, how often do you ever revisit a page that didn't load when you were interested in reading it?

Posing the question in this way reveals that performance and availability have a complicated relationship, one that involves business objectives, the user's experience, overall workload volumes, system scalability, and cost per transaction processed:

Availability

While availability is not actually a measure of performance, it certainly has an effect on performance:

Unstable systems often have specific effects on performance in addition to the other consequences of system failure. For example, instability affects the average processing time of large batch workloads, because of the possibility of failures during processing.

Additional processing is inevitable as users try to catch up with their work following a period of down time, thereby creating artificial peaks in the distribution of work, leading to higher system utilization levels and longer response times.

Highly available systems ... usually invest in redundant hardware and software and impose a need for careful design that often benefits performance. In this case availability (and any improved performance) is being traded for cost.

--High-Performance Client/Server

This is one reason why I particularly like the analysis methodology underlying the Apdex [Application Performance Index] standard. Once you have defined a response-time target, an Apdex-compliant reporting tool allocates every measurement into one of just three zones: Satisfied, Tolerating, and Frustrated. Measurements in the Satisfied zone score a full point, those in the Tolerating zone score half a point, and those in the Frustrated zone score zero. Nothing. Zilch.

For more about Apdex, see the Apdex Alliance Web site. To receive occasional updates about Apdex-related activities, you can sign up to become a supporting member. It's still free.

This post contains material first published in High-Performance Client/Server, Chapter 3: Performance Fundamentals, p61. Tags: Apdex, performance, availability, response time, download time, Performance Matters, Web performance

Why Technorati is Not Usable

Chris Loosley — Wed, 26 Sep 2007 10:30:00 +0000

I was going to write about performance and availability today, but this was not the post I had in mind. Technorati sidetracked me. So I'm going to write about Usability instead. Because Technorati provides a good counter-example -- how not to build a usable Web application that satisfies and retains customers.

In Web Usability: A Simple Framework, I described a way to think about Web site or Web application usability.

In a second post, The Dimensions of Usability, I presented the graphic shown here, and discussed the four dimensions in a bit more detail.

These four dimensions are not alternative functional goals, to be weighed against one another and prioritized. Web application effectiveness is a four-step challenge:

To satisfy customers, a Web site must fulfill four distinct needs:

Availability: A site that's unreachable, for any reason, is useless.

Responsiveness: Having reached the site, pages that download slowly are likely to drive customers to try an alternate site.

Clarity: If the site is sufficiently responsive to keep the customer's attention, other design qualities come into play. It must be simple and natural to use – easy to learn, predictable, and consistent.

Utility: Last comes utility -- does the site actually deliver the information or service the customer was looking for in the first place?

--Web Usability: A Simple Framework, October 17, 2005

As in a quiz show, to win the grand prize -- satisfied customers -- you have to get it right at every stage. Fail at any one and you will lose customers. Fail consistently at any one, and you will be out of business.

As I experienced their service today, Technorati seemed to be failing on all four fronts.

Availability

First I noticed that my browser was replacing the thumbnail portrait that usually appears (near the bottom of my sidebar) under technorati links with alternative text. Next I tried Technorati's link to Blogs that link here and, eventually, was rewarded with:

It's not that I desperately need to see my own picture there at all times, or that I think my readers are dying to see the inbound links (or blog reactions, as Technorati calls them). I know that widget in my sidebar has marginal utility -- a few people may use it occasionally. That's why I put it near the bottom, where it doesn't interfere with anyone's ability to browse the site. If it works, it does no harm. On the other hand, if it's broken, it becomes a distinct liability. Broken links lower the quality of the whole site..

In this case, the connection reset error means that the Web server accepted the request, but then took so long to respond that the browser timed out. While this may not qualify as a broken link, it had the same effect: the requested page was unavailable.

Upon checking back a few hours later, the sidebar link was working. Again, not a surprise. Intermittent outages like this have been characteristic of Technorati for a long time -- see my post on Taming the Technorati Monster.

Performance

I'm not going to dwell on this, because if you've tried to find things on Technorati lately, you already know how the service performs. For me, it typically ranges from slow to glacial, for a search engine. Maybe it's just my particular interests -- Web Performance and Application Responsiveness are not especially hot topics. Perhaps other people who are interested in more popular topics are scoring cache hits and are actually getting good Web performance. That would be ironic!

Clarity

Returning to my problems with the blog reactions widget, normally I would let this incident pass without comment. But today I actually noticed the problem while I was doing a blog search about problems with Technorati's tag indexing and search functions, because my blog seems to have fallen off their radar lately.

In the past, tags used in a post would be indexed and returned in a search within the hour, often within minutes. Today, Technorati's search function was sure I had not published anything for the last 27 days. But when I navigated manually to their page for my blog, they displayed excerpts of all my more recent posts.

Some more digging revealed that even though Technorati's site was up, and even though I could use it to navigate manually to a page listing blog reactions, the sidebar link to that same information, which Technorati's widget was generating, did not work. In any Web site or application, these kinds of internal inconsistencies are hugely frustrating. They make me doubt the accuracy and completeness of any information the application returns.

And not surprisingly, searching for help on Technorati itself does nothing to reassure me that they will be fixing these problems anytime soon. Quite the contrary, it confirms their problems, as in this amusing response:

(Is Technorati beatin' 'round the bush here? Can a search engine actually lie about what it really knows? :)

Utility

Google's Blog Search feature, however, did know something. It pointed me to this sensible post by ChristineMM:

Trying Out Blog Widgets and Tools

I would like a search box on my blog that lets my readers (and me) search for content from within the pages of my blog. The Google one that I used to use was not working right, I’d search for a word that was right in front of me or even in a blog post title and it would say there were no matches, so I dumped that function.

I then for a long time used a Technorati box. At some point I realized it was not working well either. Again I’d search for a keyword that was in a blog post title and it would say there were no matches. So this week I deleted it from my sidebar. What is the point of having a blog reader search for a topic on my blog, be told I never blogged on it, when in reality, I actually did?

...

One last thing I’ll mention is that I get a ton, and I mean a ton of blog readers through Google primarily and also some other Internet search engines. I feel that my regular use of Technorati tags helps my blog posts be found by Google and the other search engines. This drives traffic to my blog. So if you want to drive traffic to your blog, use Technorati tags in every blog post of substance.

--The Thinking Mother, September 23, 2007

Well said, Christine! If an application does not deliver the service you need, it's useless. I dropped Technorati's search box some time ago, for similar reasons. The integrated Squarespace search box is 1000 times more useful for searching the blog, and Google has the Web covered far more effectively than Technorati, in my opinion.

The bottom line

I think Christine may have homed in on the essence of the matter, sad though it may be. Unless Technorati can recover its original sense of purpose and fix its technical problems, it's not going to survive as an independent, useful, service. Perhaps its most significant contribution will be its promotion of a standard tagging format that is easily recognized and reused by other search engines.

So I'm not giving up on my Technorati tags yet, but I'm not counting on getting much value from their blog indexing or searching tools either. I've already removed their blog search and tag cloud functions from my sidebar, and their blog reactions widget is now on probation. Any more problems and it will be the next to go.

Tags: Technorati, tagging, blog reactions, problems, usability, availability, consistency, clarity, utility, Christinemm, Thinking Mother, Web performance, Performance Matters

Human Factors and Blog Design

Chris Loosley — Sat, 22 Sep 2007 07:30:00 +0000

Coding Horror

The best products are designed with Human Factors in mind. That's why Web design and usability is a frequent topic of my Web Performance Matters blog.

Jeff Atwood's blog -- Coding Horror -- focuses on programming and human factors. And according to a recent interview with Jeff on the site Daily Blog Tips, "the blog is attracting over 500,000 unique visitors every month, and it also counts 60,000 RSS readers, meaning that Jeff probably knows what he is talking about".

The Coding Horror logo was originally created to mark examples of dangerous code in the programming classic Code Complete by Steve McConnell, which Jeff rates as his "all-time favorite programming book."

I have recommended Code Complete myself. Jeff's favorite books are on my shelf too. So I respect his judgment and recommend his blog, which I have added to the blogroll on Web Performance Matters.

Thirteen blog clichés

Jeff recently published Thirteen Blog Clichés, a post summarizing his "opinions about what makes blogs work well, and what makes blogs sometimes not work so well." These are presented as a list of common mistakes to avoid (or anti-patterns). If you have a blog, or are designing one, you've probably read similar articles before. Even so, Jeff's checklist is worth a look. All such lists tend to contain a core set of common guidelines to follow and/or pitfalls to avoid, but some of Jeff's opinions step outside the conventional wisdom.

Because I maintain two blogs -- Web Performance Matters and UpRight Matters -- I decided to rate both blogs against Jeff's criteria. Here are edited versions of his recommendations, and my responses. To read Jeff's full discussions of each guideline, see the original. And for the full story, see the many responses posted by Jeff's readers in the comments section of his blog.

1. The Useless Calendar Widget

I can't think of a single time I have ever found the blog calendar widget helpful. My computer already has a calendar function, so it's not like I need another calendar displayed in my web browser.

Every post carries an obvious datestamp, so I can easily discern when it was published. But knowing whether someone posted an entry on the third Tuesday of the month? Utterly useless.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree! Someone reading a blog like Daily Kos, that publishes daily about politics or current affairs, might find a calendar useful. But a calendar isn't appropriate for our content, so we've never thought of including one. Even if we had, the Squarespace publishing platform we use (see bottom of sidebar) doesn't offer such a blog calendar widget -- another sign that it's not in great demand.

2. Random Images Arbitrarily Inserted In Text

One of the cardinal rules of web writing is to avoid large blocks of text. There are plenty of excellent web writing guides that exhort you to break up your text, using bullets, numbered lists, quotes, paragraph breaks, images -- anything to avoid creating an intimidating wall of dense, impenetrable text.

But like all good advice, (this) can be taken too far. For example, when you find yourself inserting random pictures into your writing for the sole purpose of breaking up the text. As the old adage goes, a picture is worth a thousand words. But you should no more insert a random image into your writing than you would insert a thousand random words into your writing.

Images are not glorified paragraph breaks. Images should contribute to the content and meaning of the article in a substantive way. And if they don't, they should be cut. Mercilessly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I know I am sometimes guilty of writing long posts. But I won't write a thousand words unless I have something worthwhile (I hope :-) to explain, and I try to keep all my posts interesting by breaking up the text using headings or images. And I promise that we will never include an image that bears no relationship to the subject matter!

3. No Information on the Author

Every time a reader encounters a blog with no name in the byline, no background on the author, and no simple way to click through to find out anything about the author, it devalues not only the author's writing, but the credibility of blogging in general.

Maintaining a blog of any kind takes quite a bit of effort. It's irrational to expend that kind of effort without putting your name on it so you can benefit from it. And so we can too. It's a win-win scenario for you, Mr. Anonymous.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree! That's why we provide brief introductions and longer author pages.

4. Excess Flair

Blogs work because they're simple. When we clutter up our blogs with a zillion widgets, features, and add-ons, we're destroying an essential part of what makes blogs worthwhile. Examples include "crazy" JavaScript image loading techniques, annoying pop-up image previews of links, and pictures of the last 10 visitors to your blog.

Before adding any new "feature" to your blog, consider whether its value outweighs the additional complexity it introduces.

--Jeff Atwood, Thirteen Blog Clichés [edited]

This recommendation can be controversial -- see the comments on Jeff's original post on this topic. But I agree with Jeff, and I do try to reduce the clutter whenever possible. See my decision on item 6 below, for example.

5. The Giant Blogroll

Citing your references and influences is a great and necessary thing, but obsessively listing every single blog you read is just noise. If you're really reading this many blogs, you should be linking to them organically in your blog posts, in a sort of natural quid pro quo. Wearing a giant blogroll on your sleeve is an empty gesture that feels artificial and insincere.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Agreed! On Web Performance Matters, I aim to keep my blogroll focused, and group the links into categories. We have not added a blogroll on UpRight Matters yet, but we plan to adopt the same approach.

6. The Nebulous Tag Cloud

Tagging content easily beats organizing everything into hierarchical folders, and tag categories on blogs are moderately useful, particularly for bloggers who tend to bounce around among many different topics. What I've never found useful, however, is the stereotypical tag cloud visualization, where the size of the tag word varies with its frequency.

The perception is that tag cloud visualizations are cool, like badges of honor for the tagging club. The reality is that tag cloud visualizations are chaotic, noisy, and unusable. Keep the tagging, lose the cloud. A simple sorted list of tags, along with the number of posts associated with each tag, is much more effective.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Content tagging and indexing is a complex subject, and one I have given much thought while developing our blogs. [I even read Everything is Miscellaneous, and started to write a post about it until I realized that I was just adding to the echo-chamber on that topic. See item 11 below.]

I believe that tagging with keywords has value, but the resulting folksonomy is most useful as a supplement to, not a replacement for, a carefully designed and consistently applied classification scheme or information architecture. Therefore we will continue to index our content using both methods.

However, despite investing a lot of time implementing a Technorati tag cloud for Web Performance Matters, which has been sitting in my sidebar for 4 months, I have come to the same conclusion as Jeff -- it takes up space without adding any value. So I've now removed it.

I see this as an example of the dynamic nature of blogging. It's agile publishing: you don't have to get everything right the first time. You can create something, try it out for a while, refine it, or remove it altogether. In this vein, revising your blog's layout is greatly simplified if your publishing platform is CSS-based, like Squarespace.

7. Excessive Advertisements

Advertising is a fact of life, but your blog is not Times Square. Does every square inch of whitespace have to be filled with paid links, Google AdSense, and ad banners?

Here's a related article on blog usability that's a perfect -- even ironic -- example of how you can hurt your usability with excessive, obnoxious advertising. It's everywhere.

It is almost never in the reader's interest to see advertisements, so tread very lightly, and be respectful of your audience. If you take the time to advertise responsibly, you may find that readers appreciate you for it.

Well, probably not, but it can't hurt to try.

--Jeff Atwood, Thirteen Blog Clichés [edited]

We do have a few ads. I try to organize them tastefully, so that they don't interfere with the content.

8. This Ain't Your Diary

Let's be perfectly clear: readers aren't coming to your blog to read about you. They're coming to find out what it can do for them.

That said, blogs are a place for writers to find an interested audience, and a place for readers to find a helpful peer and a unique voice. It's OK to be yourself; at some level, it is a cult of personality: people are reading not only because your content is useful to them, but because they like you.

It's normal to inject a regular dose of yourself into the conversation. But like Tabasco sauce and other powerful seasonings, a little YOU goes a long way. A really long way. Write accordingly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Agreed! I won't be writing about my experiences remodeling my house, unless I see a connection worth exploring.

9. Sorry I Haven't Written in a While

If you haven't posted anything new to your blog in a while, don't waste our time with apologies. Just write! The best apology is new and improved content. Maybe with a wee bit more consistency this time, though:

Pick a schedule you can live with, and stick to it

Don't produce substandard posts, just to keep to a schedule

Talent is far less important than enthusiasm

And the best way to demonstrate your enthusiasm -- and to improve -- is to get out there and write. Regularly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

My objectives for Web Performance Matters are much the same as when I started the blog two years ago -- to contribute an organizing framework and a regular supply of ideas. I have to admit, I've had a few long gaps in my writing. I've also apologized and promised to do better! But after reading Jeff's advice, I'm in a bind. Should I apologize for apologizing? I guess not. I'll just keep writing.

10. Blogging About Blogging

I find meta-blogging -- blogging about blogging -- incredibly boring. I said as much in a recent interview on a site that's all about blogging (hence the title, Daily Blog Tips). If you accept the premise that most of your readers are not bloggers, then it's highly likely they won't be amused, entertained, or informed by a continual stream of blog entries on the art of blogging.

Meta-blogging is like masturbating. Everyone does it, and there's nothing wrong with it. But writers who regularly get out a little to explore other topics will be healthier, happier, and ultimately more interesting to be around -- regardless of audience.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Of course, Jeff's post and this one are about blogging. But the reason we care enough to research and write about these ideas comes back to the Human Factors dimension. As we work to improve our ability to serve and communicate more clearly, we want to share what we learn to help you connect with your own readers and communities.

Cynthia writes:

Since my blogging objective is to make a competitive difference in the world, I took special note of Jeff’s post on Thirteen Blog Clichés, and of his focus on human factors. Why? Because I want to have conversations that matter -- presumably with humans!

While Chris and I stay focused on our respective blogging objectives, we work to apply the right technology to enhance understanding and to make the experience of conversing valuable for readers. Jeff’s opinions about effective technologies and techniques resonated with my blogging experience and my blogging intentions. If you have similar intentions, we think you will find them useful too.

11. Mindless Link Propagation

One of the most pernicious problems in blogging is the echo chamber effect. Most blog entries merely regurgitate what other people have said or add vapid commentary on top of news articles and press releases. Only the tiniest fraction of blog entries are original content, and only a tiny fraction of that fraction is worth your time.

If everyone knows about it, what value does that information have? My advice here is almost contrarian: if everyone else is talking about it, that means you should avoid talking about it. Switch things up. Seek out uncommon sites with unique information. If all you can find to talk about is what's already popular, you're not trying hard enough. Form your own opinion. Do your own research. Go out of your way to blaze a new trail and create something we haven't already seen hundreds of times before.

--Jeff Atwood, Thirteen Blog Clichés [edited]

That is so true! This issue is exactly what stopped me from completing my review of David Weinberger's book, Everything is Miscellaneous, even though I had read the book and dozens of pages of reviews by others. But then I found myself summarizing other readers' feedback, which amounted to a folksonomy about the topic of folksonomies -- that is, a meta-folksonomy. I began to wonder whether, if I were to review other similar discussions, and add some Technorati tags to my review, would I then be contributing to a meta-meta-folksonomy?!

People criticize Web 2.0, and the blogosphere in general, claiming that it's just a giant echo-chamber in which uninformed opinion is amplified, and true expertise is drowned out by uneducated bleating, like the sheep in Animal Farm. While that criticism may sometimes be true, it is not the whole story, and it does not do justice to the educational power of the Web.

In this case, I concluded that the world did not really need me to summarize what everyone else was saying about David Weinberger's opinions about tagging, folksonomies, and the wisdom of crowds. In fact, to write such a post would just fuel the critics' argument (and by the way, it was turning into a very long post). So I went back to writing about Web performance!

12. Top (n) Lists

Yes, exactly like this one.

Lists are a great convention. They make sense, people understand them, and they're a logical way to structure your writing. But don't let lists become a crutch. I'm always taken aback when I see the "most popular" posts on a blog dominated by Top (n) Lists. Shortcuts are only meaningful if you know what it is, exactly, you're cutting.

If you find that the Top (n) List convention is a go-to tool in your writing toolkit, consider re-balancing your writing portfolio with longer, more in-depth pieces as well. Not everything should be a sprint; throw a few small marathons in there somewhere to complement your short distance skills.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree that it's good to aim for a balance of short and longer posts. Having written technical articles for years before blogs existed, I'm actually more likely to write long essays -- see my comment on item 2 above. So to balance those marathon posts, I've found that deliberately trying to compose shorter posts whose structure is a list of guidelines or principles helps me keep that balance. My series on Performance Wisdom contains several examples of this approach.

13. No Comments Allowed

A blog without comments is not a blog. Yes, there are exceptions for massively popular blogs where comments don't scale. But until that applies, the value of the two-way conversation far outweighs any minor inconvenience on your part. It's an open secret in the blogging community that the comments are often better than the original blog entry itself. Would you browse Amazon without the user reviews?

Don't be afraid of comments. Embrace them. Moderate them. The community will respect you for it, and your blog will be better for it as well.

--Jeff Atwood, Thirteen Blog Clichés [edited]

We want comments! One of the primary reasons for blogging is to have conversations about the things that matter. It's not about us. Well, most of the time, anyway -- so we won't apologize if our writing occasionally lapses into introspection or vanity.

Note: This is cross-posted on Web Performance Matters and UpRight Matters.

Tags: human factors, Jeff Atwood, Coding Horror, blog clichés, calendar widget, random images, social bookmarks, blogroll, tagging, folksonomy, Web 2.0, tag cloud, blog advertising, meta-blogging, echo chamber, David Weinberger, Everything is Miscellaneous, Animal Farm, blog comments, blog design, Web Performance Matters, UpRight Matters

If I Had A Hammer ...

Chris Loosley — Mon, 10 Sep 2007 21:45:00 +0000

If I had a hammer
I'd hammer in the morning
I'd hammer in the evening
All over this land
I'd hammer out danger
I'd hammer out a warning
I'd hammer out love between my brothers and my sisters
All over this land

--Pete Seeger and Lee Hays, 1949 [Wikipedia]

In May 2007, after I wrote about Controlling What You Can't Measure, I had a conversation with Ben Simo (see the comments) about metrics and tools, during which I wrote:

Hammers and chisels can be very dangerous, but carpenters use them every day, and we don't brand them as "bad tools" just because some people don't know how to use them properly. Human nature being what it is, there will always be some incompetent fools who try to use a hammer and chisel to drive in a screw or open a bottle of beer, just because they have those tools handy.

A few days later, while reading a series of articles on Performance Engineering written by Scott Barber, I noticed the following quotation:

All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.

--IBM maintenance manual, 1925 [emphasis added]

This priceless piece of advice is quoted by Scott in part 9 of his 14-part series. At the time I made a note to write something about this, but after that it just sat in the "ideas for blog posts" folder for the next 3 months.

Until today, when I happened across the Ultimate Geeks Multi-Tool Hammer. Now, if I had this hammer, it turns out that I actually could use it to drive in a screw or open a bottle of beer, without being branded as an incompetent fool!

Tags: Scott Barber, Ben Simo, multi-tool, hammer, Performance Matters

Scalability is Not Optional

Chris Loosley — Fri, 07 Sep 2007 22:00:00 +0000

My recent post, Asynchronous Architectures [4], summarized a presentation by Werner Vogels at the 2007 QCON conference in London.

A subsequent post by Kent Langley in his new ProductionScale blog -- entitled Getting Rid of the Relational Database -- supports the arguments advanced by Vogels.

Describing the relational database model as "the proverbial ball and chain in the relationship between scalable applications and the underlying infrastructure," Kent writes:

The quest for seamless linear growth for technology applications is being hindered by the “elephant database.”

What would Amazon do? In a recent talk2 at QCON London Werner Vogel, the CTO of Amazon.com clearly noted that the relational database model is a essentially outdated for the needs of modern applications as a primary data storage medium. In other words, it is simply to slow and cumbersome.

Additionally, Mr. Vogel makes a critical point that in many, many cases relational databases are simply not necessary. Simple key/value pairs (hashes) are all you need.

--Joseph Kent Langley, Getting Rid of the Relational Database, August 11, 2007

Kent goes on to describe why he believes "you should break out of a one-size-fits-all way of thinking when it comes to databases, data storage, and scalable systems. Vertical scaling by throwing hardware at it is no longer sufficient for modern web scale applications".

Kent also points to Data 2.0: How the Web disrupts our relational database world, which he admits he has not read. Maybe if he had read the article he might have omitted this link.

Although the author of that article, Nitin Borwankar, supports Vogel's general conclusions, his style is to make sweeping pronouncements. He advances no technical arguments that lead up to his conclusion that "The days of Data 1.0 are past. The days of Data 2.0 are dawning, and it promises to be very disruptive for mainstream database architectures on the Web".

Proofread before publishing

I do have one small complaint. As a blogger, I know the feeling of writing down my thoughts and wanting to get them published -- right now! The short publishing cycle is one of the attractions of a blog. But if Kent could just curb his enthusiasm for long enough to proofread his posts once carefully before hitting that Publish button, his thoughts would be a lot easier to follow:

Huh?

Afterwards, I will follow up with a brief analysis or executive summary if you will of this infobit might mean for businesses. Needs commas around "if you will," and "of" should be "of what".

By way if example using the techniques of bond arbitrage Stonebraker notes quite earnestly that it is a “latency arms race.” Needs commas around "using ... arbitrage," and "if" should be "of".

So, is this inconclusive evidence of the pending death of the Relational Database? Of course not. "Inconclusive" should be "conclusive".

But, it is trend spotting in that people are again noticing that there are other ways and that those other ways just might quite faster with modern applications. "might quite" should be "might be".

--Joseph Kent Langley, Getting Rid of the Relational Database, August 11, 2007

Although I can infer what Kent is trying to say here, these glitches spoil the overall effect by forcing me to re-read his sentences to get the point. In elementary school, I learned the old English proverb: Don't spoil the ship for a ha'pworth of tar. I believe bloggers should consider readability to be as important as their message, if they want to build a faithful following.

Tags: scalability, Kent Langley, ProductionScale, Werner Vogels, QCON, relational database, Web applications, Performance Matters

Managing for Business Effectiveness

Chris Loosley — Wed, 29 Aug 2007 07:01:00 +0000

Drucker on Effectiveness vs. Efficiency

Management Wisdom: 3

There is surely nothing quite so useless as doing with great efficiency what should not be done at all

-- Peter Drucker, 1963

Peter Drucker is often called "the father of modern management". Many books and Web sites are devoted to his insights, some of which I have written about previously.

This post highlights his incisive observation about the difference between effectiveness and efficiency. I have always found it to be especially memorable, and quoted it (twice) when discussing priorities and choices in my book about software performance. Unfortunately I got the source wrong, but thanks to Google I can now correct my mistake.

It appeared in Managing for Business Effectiveness, an article in the May/June 1963 edition of Harvard Business Review ("HBR"). You can also find it in a February 2006 HBR article -- What Executives Should Remember -- a collection of excerpts drawn from HBR articles by Drucker published between 1963 and 2004.

Because Drucker's remarks are equally relevant to technical performance management and business leadership, I am cross-posting this here on Web Performance Matters, and on our new UpRight Matters blog.

Three key questions

In his 1963 essay, Drucker states that there is no magic formula, checklist, or procedure that will substitute for the hard, demanding, risk-taking work of management. But he claims that "we know how to organize the job of managing for economic effectiveness and how to do it with both direction and results. The answers to the [following] three key questions ... are known, and have been known for such a long time that they should not surprise anyone."

1. What is the manager's job? It is to direct the resources and efforts of the business toward opportunities for economically significant results. This sounds trite—and it is. But every analysis of actual allocation of resources and efforts in business that I have ever seen or made showed clearly that the bulk of time, work, attention, and money first goes to "problems" rather than to opportunities, and, secondly, to areas where even extraordinarily successful performance will have minimum impact on results.

2. What is the major problem? It is fundamentally the confusion between effectiveness and efficiency that stands between doing the right things and doing things right. There is surely nothing quite so useless as doing with great efficiency what should not be done at all. Yet our tools—especially our accounting concepts and data—all focus on efficiency. What we need is (1) a way to identify the areas of effectiveness (of possible significant results), and (2) a method for concentrating on them.

3. What is the principle? That, too, is well-known—at least as a general proposition. Business enterprise is not a phenomenon of nature but one of society. In a social situation, however, events are not distributed according to the "normal distribution" of a natural universe (that is, they are not distributed according to the U-shaped Gaussian curve). In a social situation a very small number of events—10 percent to 20 percent at most—account for 90 percent of all results, whereas the great majority of events account for 10 percent or less of the results.

-- Peter Drucker, Managing for Business Effectiveness, Harvard Business Review, May/June 1963

A principled foundation

Although Drucker writes about management effectiveness in the context of business performance, specialists in software or systems performance must ask the same questions and apply the same principles. In my book on performance management [Amazon], I described these principles as follows:

The Centering Principle: Focus on the most performance-critical components.
The Efficiency Principle: Maximize the ratio of useful work to overhead.
The Pareto Principle: Prioritize the 20% of the problem that will return 80% of the benefits.

Drucker concludes that the most crucial requirement for effective management is having ...

... the courage to go through with logical decisions -- despite all pleas to give this or that product another chance, and despite all such specious alibis as the accountant's "it absorbs overhead" or the sales manager's "we need a full product line."

This is one small example of the characteristic I find so appealing in Drucker's writing. His advice starts from an assumption that there are relevant principles, and that you can make decisions by reasoning logically from those foundations. As a mathematician, this way of looking at the world appeals to my sense of order and logic, rather than presenting me with a collection of unsupported assertions and beliefs. The HBR introduction to its review, What Executives Should Remember, sums up Drucker's appeal as follows:

Executives had come to think they knew how to run companies, and Drucker took it upon himself to poke holes in their beliefs, lest organizations become stale. But he did so in a sympathetic way. He assumed that his readers were intelligent, rational, hardworking people of goodwill. If their organizations struggled, he believed it was usually because of outdated ideas, a narrow conception of a problem, or internal misunderstandings. His insights were ... practical idea-based essays for executives, and his clear-eyed humanistic writing enhanced the magazine time and again. He helped us all to think broadly and deeply.

-- What Executives Should Remember, Harvard Business Review, February 2006

This is why so many people have enjoyed, and like Harry Joiner, continue to enjoy daily, the wisdom of Peter Drucker.

Tags: management, management principles, management wisdom, Peter Drucker, effectiveness, efficiency, Pareto Principle, 80-20 rule, Harry Joiner, UpRight Matters

Asynchronous Architectures [4]

Chris Loosley — Tue, 21 Aug 2007 07:01:00 +0000

This is the fourth in a series of posts presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment.

In a QCon conference presentation on Availability and Consistency or how the CAP theorem ruins it all, Werner Vogels, Amazon CTO, examines the tension between availability & consistency in large-scale distributed systems, and presents a model for reasoning about the trade-offs between different solutions.

I recommend you find time to watch the entire 52-minute video. The flash streaming technology that InfoQ uses is subject to buffering hiccups, and you may have to restart it a few times. So in case you want to jump to a specific section, I've assembled copies of Werner's slides, with short timestamped notes on the content of each section. Werner did not present his slides in their numbered order, so in my notes I identify slides using the numbers printed on them, not their presentation order.

Introduction

0:50: CTO's must match business with technology. Most really big IT shops must push the edge of what commercial technology can do. Technology has a very long adoption cycle -- it takes about 10 to 15 years for new technology to mature and be effective. For leading companies like Amazon, that's too slow, the scalability challenges are so great that they demand advanced solutions. So shops are forced (in effect) to do their own research, take advanced steps, just to succeed in a competitive marketplace.

2:15: Werner noted that his viewpoint disagreed strongly with that of Cameron Purdy, CEO of Tangosol, who was an advocate of database technology.

3:00: He introduced Eric Brewer's CAP theorem -- more later. [See the end of my previous post in this series, Asynchronous Architectures [3]. The CAP theorem was first propounded in a 1998 presentation -- Lessons from Internet Services: ACID vs. BASE -- by Dr. Eric Brewer of Inktomi, now a professor at UC Berkeley].

3:45: What is Scalability? [slide 2]

3:45: The meat of Werner's QCon presentation really begins here. Proportional is the key word in these definitions. Adding resources should deliver increased capacity proportional to the added resources. Or if the intent was to deliver better performance, the gains should be proportional to the added resources. Performance here is not just about response, it could mean transfering more data or larger datasets.

4:40: Another reason for needing scalability is to achieve fault-tolerance. Adding resources to achieve redundancy should not hurt your performance. Traditional technologies (like databases) won't give you this kind of scalability, because overheads increase as you scale up. These are subjects I discussed at length when explaining The Parallelism Principle in High-Performence Client/Server:

13.1 The Parallelism Principle: Exploit parallel processing

Processing related pieces of work in parallel typically introduces additional synchronization overheads, and often introduces contention among the parallel streams. Use parallelism to overcome bottlenecks, provided the processing speedup offsets the additional costs introduced.

13.2 Scalability and speed up

Scalability refers to the capacity of the system to perform more total work in the same elapsed time, when its processing power is increased.

Speed up refers to the capacity of the system to perform a particular task in a shorter time, when its processing power is increased.

In a system with linear scalability and speed up, any increase in processing power generates a proportional improvement in throughput or response time.

--High-Performance Client/Server, Chapter 13, pp383-385.

8:15: Scalability for Real Systems [slide 3]

7:00: Slide 3 is the conclusion of a cost discussion that begins before the slide is shown. The biggest threat to availability is bugs, which are a cost factor introduced by humans. So operating costs must not grow as you scale up.

8:45: But ... [slide 4]

8:30: Traditional technologies, databases, two-phase commit may work for 2-4 nodes, but they will not scale to 100's (let alone 10.000) nodes. You may not have 10,000 nodes like Amazon, but you will run into these scalability challenges at 50-100 nodes.

10:05: Principles for Scalable Service Design [slide 13]

10:05: Guidelines for services design at Amazon -- a checklist of lessons learned through hard experience:

10:05: Decentralize. Any algorithm that requires agreement will eventually become a bottleneck. Two-phase commit is an in effect an unavailability algorithm, it is guaranteed to fail as you scale up the number of participating services.
10:50: Asynchrony. Make progress under all cicumstances, even if the world is burning around you. Even if fulfillment services are burning down, you want people to be able to place orders. So work locally, don't worry about the rest of the system.
11:40: Autonomy: Each node should be able to make decisions based only on local state. If you need to reach agreement based on global conditions at high load, you are lost. Nodes may be failing, coming up, going down all the time. Probabilistic techniques work well in these circumstances.
12:35: Controlled concurrency. Reduce concurrency as much as possible, so that you do not need to use locking.
13:15: Controlled parallelism. Control traffic going to each node using careful load balancing; nodes must have spare CPU and I/O capacity so that they can do other tasks (like load re-balancing) in the background.
14:10: Symmetry. Things work really well if all nodes do exactly the same thing. It is easy to add more nodes if nodes do not have to be identified as a directory node, a data-storage node, etc. Ideally, you just install the software and run it, and it responds to any client request and does whatever task is needed, or maybe forwards the request if necessary. This is the logical way to address a requirement that I first documented in a 1993 paper, and later included in Chapter 16, Architecture for High Performance, of my book:

For a large organization, moving to an enterprise client/server system represents a major shift from monolithic systems with fixed distribution to dynamic, heterogeneous, pervasively networked environments. The next generation of systems will be an order of magnitude more dynamic--always running, always changing--with thousands of machines in huge networks.

In such an environment, content components (service providers like DBMSs) and service consumers (e.g. GUIs) must be continually added and removed.

The key to doing this is the middle tier, the hub of the three tier architecture. In the first place, this central layer acts in a connecting role to let individual clients access multiple content servers, and (of course) servers support multiple clients. A separate central tier can also:

Provide a set of services that can be dynamically invoked by both the consumer and content layers

Allow new services to be added without major reconfiguration

Allow services to be removed dynamically without affecting any participant not using those services

Allow one service provider to be replaced by another

These are all vital characteristics in a distributed computing environment.

--High-Performance Client/Server, Chapter 16, pp514-515.

15:10: Algorithms that force you to obtain agreement will become a bottleneck. So avoid using two-phase commit, maybe by denormalization to make sure your transaction always runs within a single node. Or split your task into multi-transaction workflows. You have to take an end-to-end look at the business transaction and decide. [I have always advocated this approach -- see the conclusions of the second post in this series].

17:40: You can reuse some of those principles in building teams. Small teams are best, so that each team is responsible for a well-understood piece. Team effectiveness is just as important as architectural consistency.

19:00: We call this the two pizza rule -- If you can't feed a team with two pizzas, it's too big. OK, hungry just-out-of-college students do eat more, but they work harder too! As soon as you need more than 8 people, it's hard to understand what everyone is doing. Bigger teams, of 12 or more, must have meetings, and must spend a much larger percentage of their time communicating. This discussion harks back to the famous observation by Fred Brooks:

The Mythical Man-Month

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them...

In tasks that can be partitioned but which require communication among the subtasks, the effort of communication must be added to the amount of work to be done...

The added burden of communication is made up of two parts, training and intercommunication...

If each part of the task must be separately coordinated with each other part, the effort increases as n(n-1)/2. Three workers require three times as much pairwise intercommunication as two, four require six times as much as two. If, moreover, there need to be conferences among three, four, etc., workers to resolve things jointly, matters get worse yet. The added effort of communicating may fully counteract the division of the original task.
--The Mythical Man-Month, Frederick P. Brooks, 20th Anniversary Edition pp17-18, Addison Wesley, 1995.

21:20: At Amazon, not all the services (1000 or more) support Web interactions at Amazon.com. Many are in the back-end systems such as supply-chain, fulfillment, enterprise services, handling feeds from 3rd-party suppliers, item management, recommendations, personalization.

22:15: Example of statistically improbable phrases (SIPs), an interesting digression about text analysis being implemented as yet another service. (Too long. The details are a plug for Amazon, but take time away from the main thread of the presentation).

24:30: Conclusion of the SIP discussion -- you need dependency management, and contracts. Servers can give an SLA based on workload conditions, that clients must honour. Automatic dependency discovery -- Amazon has home-grown tools that can show where dependencies exist, and the effects of failures in a network of nodes.

26:00: Scalability Through Smart System Engineering [slide 12]

26:00: Use scalable primitives. For example, RPC is not scalable. Don't conceal heterogeneity. We can pretend that systems don't fail, but in practice they do. That's the problem with RPC, it pretends to be a procedure call but it isn't, so transparency does not really work, failures do happen, performance differences do exist. So don't conceal these differences.

28:00: Configuration management. If you have 1000 services, configuration becomes really important. If your applications involve strong consistency properties, they can create problems when people leave your team.

29:00: Repair and recovery. Check out the work on Recovery Oriented Computing (at Stanford and Berkeley) by Dave Patterson and others. Can you design services to restart fast, maybe by keeping log information? If that really works well, then you can design your entire system around the principles of recovery and restart. If you don't like the behavior or performance of a service, you can kill it and let it restart. If it can be functioning again in a minute, this systems design approach can be very effective.

30:40: CAP Conjecture [slide 8]

30:40: At Amazon, all data applications are dominated by this theorem. Traditional data applications assumed that if you stored something in a database it would never go away [Durability, the "D" in the ACID properties]. In reality, because of redundancy, many nodes can be working in parallel and storing information, and then bad things can happen.

32:00: A Clash of Cultures [slide 5]

32:00: There's nothing wrong with transactions, they create a nice clean programming paradigm, and are good for programmers. But you must design for failure cases, because transactions can fail. So the ACID properties are great if you can get them, but getting these guarantees is costly. It may be fine if only a single node accesses the data. but not if 10's or 100's of machines need access.

[33:30] In that case another, more fuzzy, approach may be better -- BASE, in which data is basically available, more or less. Applications maintain a soft state, in which data is eventually consistent.

34:05: ACID vs BASE [unnumbered]

34:05: ACID has a pessimistic behavior, it will fail if it cannot reach the guarantees that you want. Availability is less important than consistency. For BASE systems, availability is the most important, and you are willing to sacrifice something to ensure it. So, for example, in a Web application you will design an application to accept and store shopping-cart input from a customer, and deal with minor problems in the data later. It's a weaker level of consistency, but you never want to tell the customer you can't accept their input.

35:50: Why the Divide? [Slide 7]

35:50: CAP stands for Consistency, Availability, and Partitioning. Eric Brewer came up with the conjecture that systems can only possess two of these three characteristics, which was subsequently proved to be true. That means systems designers must make choices, they must decide how to handle data reads and writes. If you insist on always enforcing consistency, your system may have to reject data interactions, making the system (in effect) unavailable at certain times. If you value availability and want to always accept user interactions, your application must then deal with the fact that some of its responses may later turn out to have been inconsistent.

38:45: Sometimes applications can deal with this. In the Web environment, the common technique of customer stickiness may help you to operate with lower levels of consistency. Once they have begun a session, customers are typically redirected to the same server cluster, or data center. So a local level of data consistency is sufficient; global consistency is unnecessary.

39:30: Consistency and Availability [slide 9]

39:30: Many applications have a workflow behavior. First the customer interacts with the shopping cart. At this time, availability is the most important. After that you do all kinds of things with that data, and during those activities, consistency is the most important. Now, because you are not interacting with the customer directly, if you can't obtain consistency for one data item, you can move on to process a different data item and come back later. Then you get to the shipment and delivery phase, and at this time the database is mostly read only. A data architecture that forces you to use the same powerful tool -- like a big relational database -- for all these different activities is not ideal. If you select data storage solutions that are appropriate for each phase, it's easier to scale your solution.

44:00: Partition-Tolerance and Availability [slide 10]

44:00: It's hard to program for weaker levels of consistency. Amazon has developed some API's, but Werner had no time to discuss these solutions. Slide 10 lists some examples, which he discussed briefly. The core design approach involves guaranteeing the durability of data inputs while relaxing consistency enforcement, then returning later to deal with any inconsistencies.

45:40: Techniques [slide 11]

45:40: Read the slide, because Werner does not talk about it!

We used to use a lot of DB technology at Amazon. It works really well, especially if most applications manipulate single data records using their primary keys. You can still create accessors that iterate over the entire database, but these should be relegated to a lower priority, background, status. The primary interfaces should offer only simple get/put accesses. In a production database that supports transactions, you don't need to also support queries, especially if the data is just XML text anyway. If engineers know what's inside those XML records, they may start coding against it!

48:00: Whatever DM software you are running, databases that require high performance need specialists to configure, operate, and manage them. Engineers can't do it effectively, you need DBAs, even for very simple access patterns. [Guideline 19.5 in High-Performance Client/Server: Although DBMSs may offer similar features, implementations usually differ. Never assume that a design rule of thumb learned for one DBMS can be applied unchanged to another.]

50:30: What does this mean for the data architecture?

50:30: Again, read the slide, because in the edited presentation stream, Werner appears to be speaking to a different slide altogether. And then he ran out of time, so ...

... that's it. A really insightful and informative talk by Werner Vogels. No doubt his presentation could have been improved, given more time, or even just better use of the time available. But all the same, it is very stimulating (I think) and well worth several listens, until you grasp the central points -- all of which I agree with. In fact, Werner's conclusions circle back to the conclusion of my book, and my opening statement in the first post in this series:

Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance (distributed) systems.

Documenting this talk has been both educational and satisfying. But -- since I can't type nearly as quickly as Werner can talk -- I may have misquoted him somewhere. If you spot a mistake please let me know, and I'll correct it.

This series of posts contains some material first published in High-Performance Client/Server. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

Tags: Werner Vogels, Amazon, QCon, distributed systems, asynchronous architecture, Web services, SOA, performance, scalability, synchronization, autonomy, multi-transaction, workflow, David Patterson, Recovery Oriented Computing, Fred Brooks, Mythical Man Month, Eric Brewer, ACID properties, two-phase commit, BASE, CAP theorem, performance matters

Asynchronous Architectures [3]

Chris Loosley — Wed, 15 Aug 2007 07:01:00 +0000

Dan Pritchett's Design Principle

Performance Wisdom: 13

Always assume high latency, not low latency

One of the underlying principles is assuming high latency, not low latency. An architecture that is tolerant of high latency will operate perfectly well with low latency, but the opposite is never true.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007

This is the third in a series of posts presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment.

The first reviewed the case for asynchronous communication among interdependent components or services, and Bell's Law of Waiting. The second highlighted The Fallacies of Distributed Computing, and discussed the importance of reflecting the business process in distributed systems design.

This post reviews The Challenges of Latency, an article about how asynchronous architectures can improve the quality of Web applications, published on the InfoQueue site by eBay architect Dan Pritchett in May 2007. Dan's article is especially relevant today, given the high level of interest in adopting Web services and SOA approaches.

Dan explains why global, large-scale architectures need to address latency, and what architectural patterns can be applied to deal with it. He begins by invoking the second fallacy of distributed computing:

Latency.

The time it takes packets to flow from one part of the world to another. Everyone knows it exists. The second fallacy of distributed computing is "Latency is zero". Yet so many designs attempt to work around latency instead of embracing it. This is unfortunate and in fact doesn't work for large-scale systems. Why?

In any large-scale system, there are a few inescapable facts:

A broad customer base will demand reasonably consistent performance across the globe.

Business continuity will demand geographic diversity in your deployments.

The speed of light isn't going to change.

The last point can't be emphasized enough. The speed of light dictates that even if we can route packets at the speed of light, seems unlikely, it will take 30ms for a packet to traverse the Atlantic.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Latency hurts customer service

He emphasises the connection between Internet latency and customer service:

The Internet is a part of foundation of the global economy.

Companies need to reliably reach their customers regardless of where they may be located. Architectures that force close geographic proximity of the components limit the quality of service provided to geographically distributed customers. Response time will obviously degrade the further customers are from the servers, but so will reliability. Despite the tremendous increase in the reliability of traffic routing on the Internet, the further you are from a service, the more often that service will be effectively unavailable to you.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Latency tolerance

After spelling out the principle that I have highlighed above as today's Performance Wisdom, he goes on to make the case for introducing asynchronous interactions as the way to achieve latency tolerance.

The web has created an interaction style that is very problematic for building asynchronous systems. The web has trained the world to expect request/response interactions, with very low latency between the request and response. These expectations have driven architectures that are request/response oriented that lead to synchronous interactions from the browser to the data. This pattern does not lend itself to high latency connections.

Latency tolerance can only be achieved by introducing asynchronous interactions to your architecture.

The challenge becomes determining the components that can be decoupled and integrated via asynchronous interactions. An asynchronous architecture is far more than simply changing the request/response from a call to a series of messages though. The client is still expecting a response in a deterministic time. Asynchronous architectures shift from deterministic response time to probabilistic response time. Removing the determinism is uncomfortable for users and probably for your business units, but is critical to achieving true asynchronous interactions.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Web Services in SOA - Synchronous or Asynchronous?, a paper by Torry Harris Business Solutions, offers another introduction to the pro's and con's of synchronous and asynchronous architectures.

Dan accepts that not all Web application components can be designed to function asynchronously, but argues that designers can identify those use cases that do support synchronous interactions. These arguments confirm my earlier conclusion that synchronous solutions must be combined with asynchronous designs in which the user must accept that unconfirmed changes will be reflected in the enterprise database(s) at a later time.

Data partitioning

He makes a very good point about the importance of designing data distribution from the outset:

You can decompose your applications into a collection of loosely coupled components; expose your services using asynchronous interfaces, and yet still leave yourself parked in one data center with little hope of escape. You have to tackle your persistence model early in your architecture and require that data can be split along both functional and scale vectors or you will not be able to distribute your architecture across geographies.

I recently read an article where the recommendation was to delay horizontal data spreading until you reach vertical scaling limits. I can think of few pieces of worse advice for an architect. Splitting data is more complex than splitting applications.

But if you don't do it at the beginning, applications will ultimately take short cuts that rely on a monolithic schema. These dependencies will be extremely difficult to break in the future.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

ACID or BASE?

Noting that the traditional way to maintain database consistency across partitioned data requires ACID-compliant distributed transactions and two-phase commit protocols, Dan advocates a (cleverly-named) alternative to the ACID properties, the BASE approach to database consistency:

The problem with distributed transactions is they create synchronous couplings across the databases. Synchronous couplings are the antithesis of latency tolerant designs. The alternative to ACID is BASE:
Basically Available
Soft state
Eventually consistent

BASE frees the model from the need for synchronous couplings. Once you accept that state will not always be perfect and consistency occurs asynchronous to the initiating operation, you have a model that can tolerate latency.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007

Another article worth reading, Web-Services Transactions by Doug Kaye, advances similar arguments without using the BASE terminology.

References: business-driven or event-driven architectures

While these articles present, at a high-level, a convincing case for asynchronous architectures, many others have elaborated on the implementation details. Here are five examples of more detailed treatments, in approximately descending order of generality:

The wikipedia article on Event-driven architecture.
Event-Driven Architecture Overview, a very detailed post by Brenda Michelson in her blog, Elemental Links. [A blog response from Doug McClure exemplifies the gulf between the concerns of business/application architects and those of IT/network management, despite all the talk of "alignment" these days. As I wrote in my book, "Middleware is the reason why network specialists and application programmers cannot communicate!" (Guideline 15.3, p469).]
Event-driven services in SOAby Jeff Hanson, JavaWorld.com, January 31, 2005.
Message Queuing Applications, Microsoft Developer Network (MSDN).
Developing Asynchronous Web Services with Java Message Service by Rico Cruz and Marina Sum, Sun Developer Network.

Next: the CAP theorem

Dan points out that adopting the BASE approach to consistency forces you to understand a very important principle, known as The CAP Theorem. This theorem was first propounded in a 1998 presentation -- Lessons from Internet Services: ACID vs. BASE -- by Dr. Eric Brewer of Inktomi, now a professor at UC Berkeley.

Of course there are situations where data needs to be consistent at the end of an operation. The CAP Theorem is a useful tool for determining what data to partition and what data must conform to ACID.

The CAP Theorem states that when designing databases you consider three properties, Consistency, Availability, and Partitioning. You can have at most two of the three for any data model.

Organizing your data model around CAP allows you to make the appropriate decisions with regards to consistency and latency.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Because The CAP Theorem plays a crucial role in the design of large scalable systems using asynchronous architectures, I will be devoting my next post to it.

Tags: distributed systems, asynchronous architecture, Web services, SOA, middleware, serialization, synchronization, queues, decoupled processes, multi-transaction, workflow, distributed computing, Dan Pritchett, eBay, Brenda Michelson, Elemental Links, Doug McClure, Eric Brewer, Inktomi, ACID properties, two-phase commit, BASE, CAP theorem, application performance, scalability, performance wisdom, performance matters

Asynchronous Architectures [2]

Chris Loosley — Tue, 14 Aug 2007 07:01:00 +0000

The Fallacies of Distributed Computing

Performance Wisdom: 12

The network is reliable
Latency is zero
Bandwidth is infinite
The network is secure
Topology doesn't change
There is one administrator
Transport cost is zero
The network is homogeneous

-- Peter Deutsch, James Gosling, Bill Joy, Tom Lyon

This post is the second in a series presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment. The first post reviewed the general case for asynchronous communication among interdependent components or services, and highlighted Bell's Law of Waiting.

In this post I discuss how the design of distributed systems should draw on that of manual business systems. Of course, distributed computing can shorten the timescales of some business operations enormously. But drawing analogies with the way manual systems work is an observation that will help us to design efficient and scalable distributed systems.

Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance (distributed) systems.

Starting from the wrong place

Designing a computer system based on multi-transaction workflows is not a particularly revolutionary proposal. Indeed, if we had somehow been able to skip the first 40 years of the computer age and start automating our business systems using today’s technology, we would probably not have thought it the least bit unusual, because all manual systems work this way.

But like our computers, our reasoning can be so logical that it lacks real thought. Occasionally, we need to balance our linear thinking with a small dose of simple wit like that ascribed to the Irish farmer, who, when asked by strangers for directions to a distant town, began his answer by saying “Well, I wouldn’t start from here”!

Most of our troubles stem from the fact that, when trying to reach the destination of distributed systems, we keep starting from the wrong place, namely the application designs and systems software of centralized computing:

Design discussions dwell on how best to partition applications for the distributed environment. Manual systems, however, are already partitioned naturally -- the supposedly monolithic application that is being “partitioned” would not exist in the first place if it had not been conceived as “the right solution” by a designer with a centralized computing mindset.
The reason computer science devoted so much attention to distributed databases and distributed transaction management is because these concepts are extensions of the core mechanisms of centralized information processing -- shared databases and transaction monitors.

But The Fallacies of Distributed Computing -- assembled during the 1990's by architects at Sun and the subject of today's Performance Wisdom (above) -- highlight crucial differences between centralized and distributed computing. Adding network components to an application introduces many potential problems that a centralized solution does not have to consider. So rather than trying to force the centralized mechanisms to work in a distributed environment, we should adopt mechanisms that are more appropriate.

Starting from the right place

In fact, we should start from the design of manual business systems. All large scale human systems are inherently distributed and asynchronous in nature. Even the participants in close knit team efforts operate asynchronously. We find chorus lines, cheerleaders, marching bands, and synchronized swimmers so interesting because they are such an aberration. So, before the advent of the centralized mainframe, the idea of recording an entire business transaction with a single synchronized set of human actions did not arise because it is so absurdly impossible.

Traditionally, the business process is divided into its natural components (or phases), according to the roles of the various human processors (or workers). Work flows through the phases, and information is recorded as necessary along the way. If anything goes wrong along the way, the appropriate set of compensating actions must be taken to undo whatever partial progress has been made. And the whole operation is designed to ensure that no irrevocable actions are taken too early in the process--usually meaning before the money is in the bank. Companies that mail out the diamonds before cashing the checks soon learn how to design a more effective multi-phase business process.

Asynchronous architectures may reflect manual systems

Manual systems always permit asynchronous operation of their separate components, because no other mode of operation is possible. Only computers make synchronous changes even possible. To a degree, centralized computing could deliver synchronous changes to related databases because the same computer managed all the system resources. Peaks in the workload could cause contention and delays, but provided the machine kept running, a congested machine acted as it’s own governor.

But processor technology does not allow the centralized computing model to scale up without limits. When the workload surpasses the capabilities of the largest centralized processor, the only way to keep growing is to divide and conquer -- to create a network of computers. Networked computers demand a different approach to application design. Pursuing the vision of enterprise wide synchronization of information through networked computers can cause delays and inefficiencies when any part of the whole system operates below par. This is the Achilles heel of interdependent systems--the whole is no stronger than it’s weakest part.

Therefore, the rigid concept of synchronous transactions must be replaced by a wider range of possible application designs:

The older, synchronous methods are still appropriate for changes that are within the scope of a single processor, or even for occasional communication between controlled components separated by a very carefully controlled, high speed network;
These must be combined with asynchronous designs in which the user must accept that unconfirmed changes will be reflected in the enterprise database(s) at a later time.

Business process design

When we do a good job of distributed systems design, it becomes an integral part of business process design. Rather than bending the business process to meet the needs of a centralized computer, we must blend the power of distributed computers into the business process.

Ironically, these changes bring us almost full circle back to the days of manual systems. In a manual system, it is normal for changes to be recorded quickly in one location, but for those changes to take a few days to percolate through the system, with the total processing time being somewhat uncertain. Distributed computing shortens the timescales, but applying many design principles that made manual systems work efficiently can help us to create effective and scalable distributed systems, even when we do not control the performance characteristics of all the components of those systems.

In my next post I will review some recent thinking about the intersection of asynchronous architectures and the world of Web services and SOA.

This post contains material first published in High-Performance Client/Server, Chapter 16: Architecture for High Performance, pp509-510 and 526-527. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

Tags: distributed systems, asynchronous architecture, serialization, queues, decoupled processes, multi-transaction, workflow, performance wisdom, distributed computing, fallacies, Peter Deutsch, James Gosling, Bill Joy, Tom Lyon, Sun, software performance, scalability, performance matters

Asynchronous Architectures [1]

Chris Loosley — Mon, 13 Aug 2007 07:01:00 +0000

Bell's Law of Waiting

Performance Wisdom: 11

All computers wait at the same speed

-- Dr. Thomas E. Bell, Performance of Distributed Systems, Presentation to ICCM Capacity Management Forum 7, October 1993, San Francisco

In Five Scalability Principles, I reviewed an article published by MySQL about the five performance principles that apply to all application scaling efforts. When discussing the first principle -- Don't think synchronously -- I stated that Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance (distributed) systems.

That's a quote from High-Performance Client/Server, from a section on Abandoning the Single Synchronous Transaction Paradigm, in Chapter 15, Architecture for High Performance. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

So I am planning some more posts built around excerpts from the manuscript. I'll be updating and generalizing the terminology as necessary for today's environments, and adding some guidelines in my Performance Wisdom series.

Asynchronous architectures are more scalable

The first posts will elaborate on the arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment. I begin by reviewing the general case for asynchronous communication among interdependent components or services.

In the typical distributed enterprise, there will inevitably be fluctuations in the distribution of work to be done, as business volumes rise and fall, and fluctuations in the availability of network and processing resources.

Even if we have designed our systems to accommodate peak processing volumes, it is normal for some servers, some application components, or some part of a large network to be out of action some of the time. Therefore, if we design applications that require all resources to be available before we can complete any useful work, we reduce the availability of the whole system to the level of its most error prone component.

For optimal performance, we should design applications to accommodate unexpected peaks in the workload, server outages, and resource unavailability. This means application and system design must:

Emphasize concurrent operation in preference to workload serialization
Prefer asynchronous connections to synchronous ones between clients and servers
Place requests for service in queues and continue processing, rather than waiting for a response
Create opportunities for parallel processing of workload components
Distribute work to overflow servers to accommodate peak volumes
Provide redundant servers to take over critical workload components during peaks and outages

Design applications that don’t wait

Each of these topics is too large to review in any detail in this post, but their central theme can be summed up as: Design applications that don’t wait. Note an important distinction between the behavior of individual transactions or units of work, and the behavior of the system as a whole. Individual transactions may indeed have to wait until they can obtain the processing resources they need. But the application as a whole should continue processing, with a minimal allocation of resources to any transactions flowing through the system.

That way, the scarce computing resources of one server do not sit idle waiting for delayed transactions to receive responses from other services or components. For example, here's some advice from BEA, taken from Best Practices for Application Design when programming the WebLogic Java Message Service (JMS):

Asynchronous vs. Synchronous Consumers

In general, asynchronous (onMessage) consumers perform and scale better than synchronous consumers:

Asynchronous consumers create less network traffic. Messages are pushed unidirectionally, and are pipelined to the message listener. Pipelining supports the aggregation of multiple messages into a single network call.

Asynchronous consumers use fewer threads. An asynchronous consumer does not use a thread while it is inactive. A synchronous consumer consumes a thread for the duration of its receive call. As a result, a thread can remain idle for long periods, especially if the call specifies a blocking timeout.

For application code that runs on a server, it is almost always best to use asynchronous consumers (which) prevents the application code from doing a blocking operation on the server. A blocking operation, in turn, idles a server-side thread; it can even cause deadlocks. Deadlocks occur when blocking operations consume all threads. When no threads remain to handle the operations required to unblock the blocking operation itself, that operation never stops blocking.

Bell's Law

In conclusion, when designing distributed systems, we should always recall Tom Bell's humorous observation, All computers wait at the same speed. A computing resource that is waiting to be used, especially a processor, is a wasted resource.

This post contains material first published in High-Performance Client/Server, Chapter 11: The Sharing Principle, p360. Tags: distributed systems, asynchronous architecture, serialization, queues, Bell's Law, Tom Bell, performance wisdom, software performance, scalability, performance matters

Five Scalability Principles

Chris Loosley — Fri, 10 Aug 2007 10:12:00 +0000

Five Scalability Principles

Performance Wisdom: 10

Don’t think synchronously, ...

... don’t think vertically, don’t mix transactions with business intelligence, avoid mixing hot and cold data, and don’t forget the power of memory.

-- MySQL site, 2007

The 12 Days of Scale-Out is a section of the MySQL site. It consists of a series of twelve articles, eleven of which are case studies describing large-scale MySQL implementations. But Day Six is a bit different -- it spells out five fundamental performance principles that apply to all application scaling efforts.

This subject is vitally important to MySQL, whose server replication and high availability features ... allow high-traffic sites to horizontally 'Scale-Out' their applications, using multiple commodity machines to form one logical database -- as opposed to 'Scaling Up', starting over with more expensive and complex hardware and database technology.

I know from first-hand experience that these claims are valid. At Keynote, my team used MySQL as the foundation for the Performance Scoreboard. In this data mart application, MySQL supports supports the continuous insertion of new measurements at the rate of several million per day, plus hourly aggregation into summary tables, plus the queries needed to support continually updated dashboard displays for every customer, plus any ad hoc queries generated by customers doing diagnostic investigations.

Learning the hard way

According to the article, MySQL's database experts ... have seen many companies fall into a few common traps when they first design their systems, only to run into performance issues once the explosive growth hits. So, adopting the anti-pattern approach to providing guidance, the article presents the principles as the Top Five Scale-Out Pitfalls to Avoid. These are:

Don’t think synchronously
Don’t think vertically
Don’t mix transactions with business intelligence
Avoid mixing hot and cold data
Don’t forget the power of memory

It is common for people to get smarter about performance when they have to find and fix problems. I learned about database performance first-hand between 1970 and 1995, while designing and tuning IBM database systems -- first IMS, and then DB2. In the process I discovered -- often the hard way -- that all large computer systems are subject to the same principles. And over the years I ran across other authors and teachers who (not surprisingly) had discovered the same things. Some had even invented memorable and insightful ways of describing their insights as "laws" or "rules".

When I wrote High-Performance Client/Server, I tried to capture these clever sayings as numbered guidelines. So in this post, I'm going to reprint each of MySQL's five scale-out pitfalls, followed by the corresponding guidelines and some related excerpts from the manuscript of that book (marked as "HPCL").

1. Don’t think synchronously

Thinking synchronous is the single biggest mistake in architecting a Scale-Out design. Generally, when load is added to an already-loaded system, some part of the system will become a bottleneck -- and response times will increase. In scale-out, with a large system consisting of multiple machines, thinking synchronously will add a lot of wait time and hurt performance. Any truly large scale-out design will have to introduce asynchronous communication, parallelization, and strategies to deal with approximate or slightly outdated data.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

Abandoning the Single Synchronous Transaction Paradigm

HPCL: The ... concept of a heterogeneous distributed database with synchronized updates is a vision of utopia that swims against the tide of computing technology. The tight controls over application processing that are possible on a mainframe are incompatible with many aspects of the move to widespread distributed processing... Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance enterprise client/server systems:

Decoupled processes. Decoupling occurs when we can separate the different parts of a distributed system so that no one process ever needs to stop processing to wait for the other(s). The driving force behind this recommendation is the Bell’s law of waiting: All CPU’s wait at the same speed (Guideline 11.17)
Multi-transaction workflows. Often, we can split up the business transaction into a series of separate computer transactions. We call the result a multi-transaction workflow. The motivation for this recommendation is the Locality Principle, Group components based on their usage (Guideline 10.1), and (one corollary of) the Parallelism Principle, Workflow parallelism can lower perceived response times (Guideline 13.7)

--High-Performance Client/Server, Chapter 16: Architecture for High Performance, pp506-507

2. Don’t think vertically

It's a mistake to think that a system can be grown by scaling vertically, that is, by buying bigger machines with more CPUs. Throwing more power at an existing implementation -- which is probably synchronous and most likely already suffering from lock waits -- is only going to make it 'wait faster'. By planning for horizontal scale-out, almost from the start, a business is already planning in the direction of distributed, asynchronous systems, which will make it easier to add more capacity later on.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

HPCL: In Chapters 11 and 12 we discussed the performance of shared resources, and how to minimize the delays caused by bottlenecks. When there is excessive demand for a single shared resource, one way to break the logjam is to divide and conquer. Dennis Shasha’s database design principle, Partitioning breaks bottlenecks [1], concisely expresses this motivation for processing work in parallel.

[1] Dennis E. Shasha in Database Tuning: A Principled Approach, Prentice-Hall, 1992, p3

The idea behind parallelism is simple: take several items of work and process them at the same time. Naturally, this is faster than processing the same work serially. But while it may be an obvious and attractive design technique, processing in parallel does introduce new problems of its own. Not all workloads can be easily subdivided, and not all software is designed to work in parallel. Processing related pieces of work in parallel typically introduces additional synchronization overheads, and in many situations produces new kinds of contention among the parallel streams. Connie Smith’s version of the Parallelism Principle recognizes these complications: Execute processing in parallel (only) when the processing speedup offsets communication overhead and resource contention delays [2].

[2] Connie U. Smith, Performance Engineering of Software Systems, Addison Wesley, 1990, p55

The Parallelism Principle

To sum up, Smith’s and Shasha’s two views nicely highlight the dilemma facing every designer when considering parallelism. Combining them gives us The Parallelism Principle (Guideline 13.1), Exploit parallel processing:

Processing related pieces of work in parallel typically introduces additional synchronization overheads, and often introduces contention among the parallel streams. Use parallelism to overcome bottlenecks, provided the processing speedup offsets the additional costs introduced.

--High-Performance Client/Server, Chapter 13: The Parallelism Principle, pp382-383

3. Don’t mix transactions with business intelligence

Many large systems are OLTP systems that do not have a data export phase inside the application. Therefore, they oftentimes contain vast amounts of business intelligence data. If an OLTP system, for the same number of users/articles/orders, grows over time, then it likely has a data warehouse or data mart struggling to get out. Separating the data onto different databases and/or servers will go a long way in improving performance for both the transactional application and analytic operations.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

HPCL: How are we to reconcile an immediate need to process large decision-support queries with an ongoing transaction workload that demands high throughput and short response times? The more resources consumed by queries, the fewer remain to process transactions, and the larger the impact of the query workload on transaction throughput.

On the other hand, if we try to maintain overall throughput by artificially restricting the resources allocated to queries, queries take much longer. This alone can cause political problems, unless expectations have been set properly. But there is a potential side effect that is even more damaging. If the DBMS holds any database locks for a long running query, subsets of the transaction workload may have to wait until the query completes, causing erratic transaction response times.

Inmon’s rule

The usual resolution of this dilemma is to avoid mixing short transactions with long-running queries in the first place. I refer to this as Inmon’s rule because the well known speaker and author Bill Inmon spent many years during the 1980s evangelizing this concept, making the rounds of the database user groups, talking about the importance of separating operational from informational processing.

Don’t mix short transactions with long-running queries. When we have high-performance operational workloads, we should keep them on a separate processor, separate from ad hoc or unknown queries, which may have massive processing requirements. (Guideline 10.20)

--High-Performance Client/Server, Chapter 10: The Locality Principle, pp310-311

4. Avoid mixing hot and cold data

Similar to #3 is mixing hot (frequently-changed) and cold (more static) data, especially when it comes to write activity. Since database writes are more difficult and expensive to scale, it is advisable to keep this type of data away from data that does not change that often. Again, separating the data onto different databases and/or servers can significantly enhance your application's performance.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

This one is interesting. In Chapter 12 on Database Locking, I discussed database hot spots:

HPCL: Although locking is essential for maintaining data integrity, excessive contention can occur when too many concurrent applications need to lock the same data item, or the same small set of data items. Database designers call this type of locking bottleneck a hot spot. Because several applications are reading and writing to the same portion of a database, it is quite common for deadlocks to be caused by hot spots. A hot spot can arise for one of three reasons:

Natural hot spots exist in the business data. It is very rare for activity to be evenly distributed across all areas of the business--recall the Centering Principle (Guideline 4.9. Think globally, focus locally. This guideline is related to the Pareto Principle, or 80-20 rule.). Sometimes, thanks to a highly skewed distribution of work, a large percentage of database updates apply to a small fraction of the business data. ... Automating these types of business applications tends to create database hot-spots naturally, unless we are careful to design the databases and the associated processing to eliminate them.
The application’s design creates artificial hot spots. When applications are designed to maintain the current value of a derived statistic like a sequence number, a total, or an average, instant hot-spots are created in databases because every instance of a program must read and update the same data item.
Locking protocols against physical data structures create artificial hot spots. Even though applications are not manipulating the same data, they may well be reading and writing to the same physical data or index pages, which will cause contention if page level locking is being used. Common examples of this type of problem occur when all the rows in a table fit on a small number of pages, or when data is inserted sequentially on the last page of a table based on a time or a sequence number.

--High-Performance Client/Server, Chapter 12: Database Locking, pp373-374

All my recommendations (pp 374-380) involved ways to reduce database contention by spreading out the data elements that comprise the hot spot. In contrast, the MySQL suggestion is to separate hot (frequently-changed) and cold (more static) data altogether, and then to focus on special tuning to improve the performance of the hot data. While I did not propose this approach, I believe it does not conflict with my own advice. As I noted when discussing Inmon's rule, Inmon’s rule is not just for transactions and queries. It can be applied to any mix of workloads that have different performance characteristics.

5. Don’t forget the power of memory

Data accessed in memory produces infinitely better response times than the same data accessed on disk. Once the most-often referenced/accessed data -- oftentimes called the "working set" -- exceeds the amount of available memory, a system runs the risk of becoming disk-bound, with the end result being poorer performance. When designing a Scale-Out architecture, care must be taken not to exhaust a single server's memory allocations so that it becomes disk bound. Instead, an application's working set must be smartly divided among the servers participating in a scale-out design so that data is always accessible in RAM.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

Substitute faster devices for slower ones

HPCL: The table (below) shows the typical hierarchy of computing resources found in an enterprise client/server environment, organized by relative speed with the fastest at the top. To improve an application’s performance, we must look for design changes that will make its resource usage pattern migrate upwards in the hierarchy.

Device Type	Typical Service Time	Relative to 1 second
High Speed Processor Buffer	10 nanoseconds	1 second
Random Access Memory	60 nanoseconds	~6 seconds
Expanded Memory	25 microseconds	~1 hour
Solid State Disk Storage	1 millisecond	~1 day
Cached Disk Storage	10 milliseconds	~12 days
Magnetic Disk Storage	25 milliseconds	~4 weeks
Disk via MAN/High Speed LAN Server	27 milliseconds	~1 month
Disk via Typical LAN Server	35-50 milliseconds	~6-8 weeks
Disk via Typical WAN Server	1-2 seconds	~3-6 years
Mountable Disk/Tape Storage	3-15 seconds	~10-50 years

Substitute memory and processor cycles for disk I/O

Database caching or buffering reduces the cost of re-reading frequently reused portions of a database by retaining them in memory, improving responsiveness by trading off memory for processor resources and disk I/O. Of course, the costs of searching for data in cache are all wasted overhead when the data isn’t actually there. This situation is termed a cache miss. The relatively small overhead of cache misses must be weighed against the larger savings we get whenever there is a cache hit. Typically, no matter how large the cache, cache hits consume fewer processor cycles and a lot less time than would the corresponding disk I/O.

--High-Performance Client/Server, Chapter 14: The Trade-off Principle, pp432-433

Parallel processing can be done on machines with SMP, MPP, or various hybrid architectures ... Regardless of the hardware architecture, the objective is to increase processing power by adding more processors. But depending upon both the workload and how the processors and other devices are interconnected, other hardware constraints, for example, the memory or I/O bus, can prevent the full exploitation of the additional processors. If some other component is the bottleneck, we obtain no benefit from adding more processors--in fact, we may even make the bottleneck worse.

One solution is to remove the constraint by giving each processor its own memory, and/or its own disk subsystem. This solution, however, complicates system software development. For example, database software must coordinate the contents of in memory data caches across the different processors if each processor has its own dedicated memory. Removing hardware constraints may simply move the problem to the software, unless the software has been specially written to overcome these problems.

--High-Performance Client/Server, Chapter 13: The Parallelism Principle, pp417-418

Your contributions ...

I've included just a few extracts from my book that came to mind as I was compiling this post. If you know of other informative discussions of the same principles, in a book or on the Web, please post a comment below sharing your knowledge. Of course, questions or observations of your own are also welcome.

Tags: scalability, MySQL, database, performance, performance wisdom, Data mart, Dennis Shasha, Connie Smith, Parallelism, hot spot, Bill Inmon, Pareto Principle, 80-20 rule, working set, cache, response time, computer memory, Performance Matters, application performance

Latency, Bandwidth, and Response Times

Chris Loosley — Tue, 24 Jul 2007 10:00:00 +0000

Latency, Bandwidth, and Station Wagons focused primarily on the limitations of network bandwidth, and the time required to transmit massive data volumes. While that is an interesting topic, and one that produces some surprising results (like the fact that FedEx is still faster than the Internet), it is not particularly relevant to the subject of Web performance, which depends on the time required to transmit many small files.

My post highlighted It's Still The Latency, Stupid, by William (Bill) Dougherty in edgeblog. Bill's title pays homage to a famous 1996 article by Stuart Cheshire about bandwidth and latency in ISP links, It's the Latency Stupid.

Over a decade later, Bill points out, Cheshire's writings are still relevant: One concept that continues to elude many IT managers is the impact of latency on network design ... Latency, not bandwidth, is often the key to network speed, or lack thereof. This is especially true when it comes to the download speeds (or response times) of Web pages and Web-based applications. In this post I explain why, providing some supporting references and examples to support my argument.

The edgeblog debate

Supporting Bill's point about the lack of understanding of this fact, Simon Howard (posting as fragglet) responded with It's the bandwidth, stupid, in which he disagreed strongly with Bill's post. A lively discussion ensued in the comments on edgeblog, beginning here.

Bill then followed his first post with Part 2, which discussed four possible tuning actions to reduce the impacts of network latency:

Tweak the host TCP settings
Change the protocol
Move the service closer to the user
Use a network accelerator

This post was also disputed by fragglet, prompting another response from Bill in the edgeblog comments, here. These discussions are an excellent illustration of the kinds of misunderstandings that exist about the role of network latency as a determinant of performance. Perhaps the most revealing exchange is this one:

Fragglet: You are saying that latency causes network problems and that by improving latency you can improve your network. I assert that this is false. If you have latency problems, they are a symptom of network congestion. If your network is suffering from serious congestion, it probably needs more bandwidth.

Bill: Wow. It is impressive how someone can miss the point so completely so many times. While network congestion will add to latency, latency is in and of itself a problem. In a network with zero congestion, latency will still be a problem. The problem is distance. More bandwidth cannot improve upon the speed of light. Sorry. This is the whole point of my article. Latency does cause issues unrelated to bandwidth or congestion. Those issues can be reduced with planning.

Indeed! I will now explain why fragglet is wrong and Bill is right. As promised (at the end of my station wagons post) I am going base my explanation on the 2001 article by Alberto Savoia, Web Page Response Time 101 [**], which I introduced previously. That's because even though his examples are a bit dated (how many people are still using a 28.8Kbps modem today?), Alberto's article provides a concise and readable explanation of the technical principles involved.

[** Warning: even though this reprint is just 6 pages, it's a 2.5Mb pdf, so wait until you're on a fast connection].

The page response time formula

The Complete Formula

R = 2(D+L+C)+(D+C/2)((T-2)/M)
+Dln((T-2)/M+1)+max(8P(1+OHD)/B,
DP/W)/(1-sqrt(L))

B = Min line speed (bits per second)
C = Cc + Cs
Cc = Client processing time (seconds)
Cs = Server processing time (seconds)
D = Round trip delay (seconds)
L = Packet loss (fraction)
M = multiplexing factor
OHD = Overhead (fraction)
P = Payload (bytes)
R = Response Time (seconds)
T = application turns (count)
W = Window size (bytes)

The key to this issue is a simple formula for Web page download time. Credit for the original research goes to Peter Sevcik and John Bartlett of NetForecast Inc.. Their 2001 research report, Understanding Web Performance, was published in Business Communication Review (BCR) in October 2001, and can also be downloaded as a pdf. It explains Web page response time using "The Complete Formula" shown here.

Alberto's contribution was in simplifying this formula to the version shown in his article's Figure 1, reproduced below. He notes that:

This formula makes several generalizations and assumptions, and its accuracy varies through the possible range of values (it tends to overestimate below eight seconds and underestimate over eight seconds).

Actually, even that explanation hides some further assumptions. His reference to eight seconds involves assumptions about typical connection latency and bandwidth, typical Web page sizes, and the typical number elements (separately downloadable files) that make up a Web page.

But, as he says, for the purposes of this article, however, it will do just fine, since it introduces the key variables that impact page response time and shows you how they relate to each other, without introducing excessive complexity. Alberto describes the six key variables as follows:

The six parameters of Web response time

Page size: Page size is measured in Kbytes, and on the surface, the impact of this variable is pretty obvious: the larger the page, the longer it takes to download. When estimating page size, however, many people fail to consider all the components that contribute to page size—all images, Java and other applets, banners from third sources, etc.—so make sure you don’t overlook anything.

Minimum bandwidth: Minimum bandwidth is defined as the bandwidth of the smallest pipe between your content and the end user. Just as the strength of a chain is determined by its weakest link, the effective bandwidth between two end points is determined by the smallest bandwidth between them. Typically the limiting bandwidth is between the users and their ISPs.

Round trip time: In the context of Web page response time, round-trip time (RTT) indicates the latency, or time lag, between the sending of a request from the user’s browser to the Web server and the receipt of the first few bytes of data from the Web server to the user’s computer. RTT is important because every request/response pair (even for a trivially small file) has to pay this minimum performance penalty. As we shall see in the next section, the typical Web page requires several request/response cycles.

Turns: A typical Web page consists of a base page [or index page] and several additional objects such as graphics or applets. These objects are not transmitted along with the base page; instead, the base page HTML contains instructions for locating and fetching them. Unfortunately for end-user performance, fetching each of these objects requires a fair number of additional communication cycles between the user’s system and the Web site server—each of which is subject to the RTT delay I just mentioned.

Server processing time: The last factor in the response time formula is the processing time required by the server and the client to put together [i.e. generate and render] the required page so it can be viewed by the requester. This can vary dramatically for different types of Web pages. On the server side, pages with static content require minimal processing time and will cause negligible additional delay. Dynamically created pages (e.g., personalized home pages like my.yahoo.com) require a bit more server effort and computing time, and will introduce some delay. Finally, pages that involve complex transactions (e.g., credit card verification) may require very significant processing time and might introduce delays of several seconds.

Client processing time: On the client side, the processing time may be trivial (for a basic text-only page) to moderate (for a page with complex forms and tables) to extreme. If the page contains a Java applet, for example, the client’s browser will have to load and run the Java interpreter, which can take several seconds.

--Alberto Savoia, Web Page Response Time 101, STQE, July/August 2001
[Minor clarifications added]

There is no question that these six variables do account for Web page response times, and do operate in the directions prescribed in the formula. As one confirmation, consider the authoritative textbook on the quantitative aspects of computing, Computer Architecture by John L. Hennessy and David A. Patterson ("H&P"). In the 3rd Edition (2003), Chapter 8 covers networks, and page 798 gives this simple formula for the total latency of a message:

Total latency =
Sender overhead + Time of flight + Message size/Bandwidth + Receiver overhead

The Web uses TCP, which transmits data as message segments comprising one or more packets, with each segment being acknowledged. So we can view a Web page download as a succession of H&P's message transmissions. Because each TCP segment is followed by an acknowledgment, H&P's time of flight variable corresponds to Alberto's round-trip time. And so summing N instances of H&P's formula produces Alberto's formula, with Turns having a value of N.

This demonstrates that the formula is correct, but if we wanted to use it to predict page download time, the task of selecting the right values to plug into those variables involves some complications.

Turn, turn, turn ...

Alberto does explain how to use the formula. But in practice, I would expect those directions to produce a worst-case estimate for response time, mainly because the value of the Turns variable should probably be lower than Alberto's method suggests. Three factors affect the estimation of turn counts:

Alberto assumes HTTP 1.0, but HTTP 1.1 is now the predominant platform. HTTP 1.1 implemented persistent TCP connections, so that browsers do not repeatedly close and reopen TCP connections with a server, and (for large files) TCP segments will be larger, because of the way TCP slow-start operates. So most browsers will now download a Web page using fewer turns.
Browsers can open up to two parallel connections for each distinct server domain, removing some turns from the synchronous response time path.
Most users of most Web pages already have some page elements in their browser caches, eliminating the turns that would be needed to fetch them. On the other hand, recent research at Yahoo suggests that browser caching benefits may be less than imagined -- see Performance Research, Part 2: Browser Cache Usage - Exposed! by Tenni Theurer.

Fortunately, such complications do not prevent us from using the formula to explain the relative importance of bandwidth and latency.

Implications of the formula

To demonstrate this, consider a newer version of the formula shown in Figure 2 below. This comes from a September 2006 NetForecast report, Field Guide to Application Delivery Systems. The only difference between this and Alberto's version is in its use of curly equals, signifying "is approximately equal to":

Figure 2. Response Time Formula by NetForecast

The NetForecast paper goes on to discuss how each of the six factors affects response times, using Figure 3 (below) to summarize its points.

Figure 3. Response Time Causes and Effects

In addition to the issues highlighted by NetForecast, we can evaluate the relative contributions of Bandwidth and Latency to overall response time by simply comparing the two factors [Payload/Bandwidth] and [Turns x RTT] in typical Web environments.

[Payload/Bandwidth]: Alberto's paper focuses on dial-up performance, using the example of effective bandwidth being 4Kbytes/sec. In this situation, a 120K page may take 30 seconds to transfer, before we even consider any latency effects. But as bandwidth increases, this factor becomes progressively smaller. A broadband connection of 1.5Mbps can transfer about 150Kbytes/sec, reducing this factor to 0.8 secs for a 120K Web page, or 2 secs for a 300K page.
[Turns x RTT]: According to NetForecast, the average Keynote Business 40 home page requires 60 turns and 300 kilobytes to load. If RTT (or "ping times") are in the range of 100-200ms (typical times for many consumers), 60 turns will add 6-12 secs for network latency. Ping times will certainly be faster for broadband connections than they are for dial-up, because of all the extra analog-digital conversion delays imposed by the dial-up technology. But at a certain point, the speed of light and the latencies of the carriers' network devices (hubs, routers) limit further improvement.

The bottom line ...

In the analysis above, I showed that as connection bandwidth increases, the effect of the first factor in the response time formula approaches zero. No such scaling effect exists for the second factor -- neither latency nor turns can ever be made to approach zero. In fact, as Web sites and applications keep growing in sophistication, turns are increasing. The NetForecast paper states that over the past decade turn counts for the Keynote Business 40 Web sites have grown 12% per year and payload has grown 20% per year.

Even the newest AJAX technology may increase, not reduce, turns, as designers attempt to replace large monolithic file downloads with multiple smaller requests. Unless those smaller requests can also be designed to happen asynchronously, while the user is doing something else, they will only add to the delay due to network latency.

This is why I agree with Bill Dougherty that It's Still The Latency, Stupid! And until someone finds a way to move bits faster than the speed of light, that's not going to change.

Tags: William Dougherty, edgeblog, Simon Howard, fragglet, performance, bandwidth, throughput, latency, ping time, round-trip time, Stuart Cheshire, Web performance, Web application, download time, Alberto Savoia, Peter Sevcik, John Bartlett, NetForecast, Fedex, Royal Pingdom, station wagon, Performance Matters

Java Performance Optimization

Chris Loosley — Fri, 20 Jul 2007 05:55:00 +0000

Do you subscribe to email newsletters? If you're like me, you get lots of them. New ones appear in my inbox every morning. They pile up, demanding to be read. In fact, they seem to breed like rabbits, producing new offspring -- when did I express an interest in Enterprise VOIP Security Architecture issues? Sometimes in a housekeeping splurge I delete a few dozen at once, suffering a momentary twinge of anxiety at having perhaps missed something important. So usually I skim them before hitting the delete button.

TechTarget's Search Software Quality service seems to be especially prolific, but is also a regular source of interesting references -- like TheServerSide.com, the subject of a recent note. According to the site's home page:

Java Performance Management for Large-Scale Systems

There are many classes of enterprise applications that have stringent performance and scalability requirements. TheServerSide.com has assembled a collection of resources to help you better design, develop, test and manage high performance, large-scale systems - learn new and innovative approaches for performance tuning, memory management, concurrent programming, JVM clustering and more.

Pro Java EE 5 Performance Management and Optimization

One of the articles listed on TheServerSide.com was a book review of Pro Java EE 5 Peformance Management and Optimization by Steven Haines:

Pro Java EE 5 Performance Management and Optimization features proven methodology to guarantee top-performing Java EE 5 applications and explains how to measure performance in your specific environment. The book details performance integration points throughout the development and deployment lifecycles. For QA and preproduction stages, author Steven Haines guides the reader through testing and deploying Java EE 5 applications with a focus on assessing capacity and discovering saturation points. Haines also defines the concept and application of wait-based tuning.

In addition, the book explains assessing and improving the health of applications upon deployment. The topics covered include trending, forecasting, and capacity assessing and planning. Haines also walks through the creation of a formal Java EE 5 Performance Management Plan, customized to an environment to help interpret and react to changing trends in usage patterns.

Published by Apress, May 2006. ISBN: 1-59059-610-2

The review includes the option to download the full text of two chapters:

Chapter 6: Performance Tuning Methodology

... focuses on setting up a proper testing environment and explores the concept of wait-based tuning. Haines explains the steps necessary to implement a formal performance tuning methodology and guides the reader through a complete tuning example -- [pdf].

Chapter 9: Performance and Scalability Testing

... discusses the difference between the concepts of performance and scalability, and outlines the strategy of ensuring performance before testing for scalability. Haines also leads a detailed exploration into the ultimate scalability test - the capacity assessment - and explains how to assemble a formal Capacity Assessment Report -- [pdf].

Steven Haines

Steven Haines appears to be well qualified to write on this subject. If you'd like more background, there's an interview on the JavaPerformanceTuning.com site. And here's a publisher's bio:

Steven Haines is the author of three Java books: The Java Reference Guide (InformIT/Pearson, 2005), Java 2 Primer Plus (SAMS, 2002), and Java 2 From Scratch (QUE, 1999). In addition to contributing chapters and coauthoring other books, as well as technically editing software publications, he is the Java Host on InformIT.com. As an educator, he has taught all aspects of Java at Learning Tree University as well as at the University of California, Irvine. By day he works as a Java EE 5 Performance Architect at Quest Software, defining performance tuning and monitoring software as well as managing and performing Java EE 5 performance tuning engagements for large-scale Java EE 5 deployments, including those of several Fortune 500 companies.

Also, SAMS Publishing has a three-part article he wrote in 2003 about J2EE performance tuning -- part 1, part 2, and part 3.

Other excerpts ...

For more samples of the book's content, an article published on the JavaWorld site seems to contain most or all of Chapter 14, Solving Common Java EE Performance Problems. While Chapter 6 addresses the principles of performance tuning, this long article (13-parts) goes into a lot more technical detail. It concludes:

Solving common Java EE performance problems

While each application and each environment is different, a common set of issues tends to plague most environments. This article focused not on application code issues, but on the following environmental issues that can manifest poor performance:

Out-of-memory errors

Thread pool sizes

JDBC connection pool sizes

JDBC prepared statement cache sizes

Cache sizes

Pool sizes

Excessive transaction rollbacks

In order to effectively diagnose performance problems, you need to understand how problem symptoms map the root cause of the underlying problem. If you can triage the problem to application code, then you need to forward the problem to the application support delegate, but if the problem is in the environment, then resolving it is within your control.

--Steven Haines, JavaWorld.com, June 19, 2006

Finally, the Google Books site also contains about 25 shorter excerpts, each 3 or 4 pages, that will give you a really good picture of what the book covers and how Steven approaches his subject matter.

I'd be rather surprised if, after browsing all these informative and well-written samples, you don't decide that you need to order a complete copy for yourself. I have!

Tags: Steven Haines, Quest, Apress, TechTarget, Java, JavaWorld, TheServerSide,
performance tuning, performance, tuning, scalability, Performance matters, Web performance

Latency, Bandwidth, and Station Wagons

Chris Loosley — Fri, 13 Jul 2007 19:05:00 +0000

One concept that continues to elude many IT managers is the impact of latency on network design. 11 years ago, Stuart Cheshire wrote a detailed analysis of the difference between bandwidth and latency in ISP links [It's the Latency Stupid]. Over a decade later, his writings are still relevant. Latency, not bandwidth, is often the key to network speed, or lack thereof.

That's from It's Still The Latency, Stupid by William (Bill) Dougherty, writing in edgeblog on May 31, 2007. Bill follows that opening paragraph with a very readable explanation of the vital importance of latency (round-trip time) as a factor affecting performance in TCP networking. He uses what he calls the Sandbag Problem to illustrate his points:

Shifting a Heap of Sand

Let’s say the two of us are trying to fill sandbags. My job is to scoop sand into a container and hand the full container to you (data). Your job is to empty the container into a sandbag and hand the empty container (ACK) back to me. Occasionally you drop the container so I have to fill it again (Retransmit). If we were standing next to each other, the time it takes for me to hand the container to you, have you empty it, and hand it back to me (latency) would be very small. Now imagine there is a 6′ wall between us, and I need to hand the container over to you.

The wall changes several aspects of our filling operation. First, the size of the container must be smaller because I cannot lift the same weight over my head that I can lift at waist level. Second, the time to complete one cycle would increase because it takes longer to lift the container 6′ than it does 3′. Third, you would drop more containers so retransmissions would increase. As the wall gets taller, the problem gets worse. If the wall were 10′ tall, we would be throwing containers instead of lifting them, so they would need to be even smaller. The containers would be traveling 20′ round trip instead of 12′ so the delay would increase 75%. And we would need to send a lot more containers to move the same amount of sand.

--William (Bill) Dougherty, It's Still The Latency, Stupid, edgeblog, May 31, 2007

This is a topic that anyone who cares about the performance of Web-based applications needs to understand, because it is the key to most performance optimization and tuning initiatives. And since both Bill's post and the ensuing discussion in the readers' comments are very educational, I decided immediately to write something linking to it. But because I wanted to tie in some other ideas, and discuss Web Page Response Time 101, a paper written in 2001 by Alberto Savoia (see Four Laws of Web Site Performance), I saved a reminder in a draft post until I had time to focus on it. That was a month ago.

Well, once you drop an idea in your mental in-tray, it starts to search for its proper place in the files. Connections pop up all over the place. Sure enough, once I had made a mental note to write about that edgeblog post, I began to notice other blog posts on the same topic. A couple of weeks later, I saw a post about Latency vs. Throughput by Steve Harris in his DSO Guy blog. Instead of people moving sandbags, Steve uses trains and planes shipping coal supplies in his example:

Shipping a Load of Coal

Imagine you have to move a bunch of coal across the country and deliver it to a coal processor. Now say that on the west coast, the receiver of the coal can process 100 units of coal an hour. You have 1 train that can haul 10,000 units of coal and takes 48 hours to get to its destination. You have 1 plane that can deliver 100 units of coal in 12 hours.

If the most important thing was to have the coal soon, then the plane is faster (lower latency). But, if the most important thing is to have the coal-processing pipeline filled on the west coast over time then train is faster (higher throughput).

--Steve Harris, Latency vs. Throughput in DSO Guy, June 14, 2007

Then last week I ran across this amusing item in the Royal Pingdom blog. Shunning such analogies as sandbags or coal sacks, it gets straight to the point:

FedEx still faster than the Internet

Imagine a company with two offices in different cities, perhaps even in different countries. Each office has a 100 megabit internet connection. If the company needs to send a large amount of data from one office to the other, theoretically a 100 megabit connection can muster about 45 gigabyte in one hour if there are no bottlenecks on the way. This ends up being just over one terabyte of data in 24 hours.

In other words, for anything larger than one terabyte, it would be faster for this company to just send the data on disks for over-night delivery.

--Royal Pingdom, April 11, 2007

And once I started actually looking for connections, I found this post by Jonathan Schwartz, CEO and President of Sun:

Moving A Petabyte of Data to Hong Kong by Sailboat

I made a speech last week at which I asserted it was faster to send a petabyte of data from San Francisco to Hong Kong by sailboat, than by the internet.

I got quite a few "how can that possibly be true?" kinds of questions, so here's the math. (Full disclosure, I am a mathematician by training, which guarantees me a lifetime of small "off by one" errors in all subsequent calculations - so if I get something wrong, be gentle).

A petabyte is a thousand terabytes, which is a million gigabytes, or a billion megabytes. Or 8 billion megabits. With me so far?

So if you had a half megabit per second internet connection, which is relatively high in the US (relatively low compared to residential bandwidth available in, say, Korea), it'd take you 16 billion seconds, or 266 million minutes, or 507 years to transmit the data. Can you sail to Hong Kong faster than that? At a full megabit, just divide the time in half. Even at a hundred megabits (about the highest, generally available, of any carrier I've seen), it's a few years.

--Jonathan Schwartz, Moving A Petabyte of Data, Jonathan's Blog, Mar 12, 2007

If you've been following my links and reading more than my excerpts alone, you've probably spotted a common thread running through these articles and the comments. I already knew about the station wagon analogy, and so I was looking for it.

Jumping on the band(width station)wagon

All these writers are following an already well-trodden path, and their conclusions echo a well-known observation first made by Andrew S. Tanenbaum in his book, Computer Networks:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway

Many writers quote this, for example Tony Dye in Station Wagon Bandwidth. Some then go into more detail; a humorous post on Everything.com calculates the bandwidth of a station wagon at 13 petabytes/second. Theo Lagendijk in Theo's Blog has the full context of the saying:

An industry standard Ultrium tape can hold 200 gigabytes. A box 60 x 60 x 60 cm can hold about 1000 of these tapes, for a total capacity of 200 terabytes, or 1600 terabits (1.6 petabits). A box of tapes can be delivered anywhere in the United States in 24 hours by Federal Express and other companies. The effective bandwidth of this transmission is 1600 terabits/86,400 sec, or 19Gbps. If the destination is only an hour away by road, the bandwidth is increased to over 400Gbps. No computer network can even approach this.

For a bank with many gigabytes of data to be backed up daily on a second machine (so the bank can continue to function even in the face of a major flood or earthquake), it is likely that no other transmission technology can even begin to approach magnetic tape for performance. Of course, networks are getting faster, but tape densities are increasing, too.

If we now look at cost, we get a similar picture. The cost of an Ultrium tape is around $40 when bought in bulk. A tape can be reused at least ten times, so the tape cost is maybe $4000 per box per usage. Add to this another $1000 for shipping (probably much less), and we have a cost of roughly $5000 to ship 200TB. This amounts to shipping a gigabyte for under 3 cents. No network can beat that. The moral of the story is:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

—Andrew S. Tanenbaum, Computer Networks. Prentice-Hall, 1996

For more detailed discussions of the price/performance of Sneakernets, an article by Jeff Atwood (plus many reader comments) on the blog Coding Horror reviews some contributions by Jim Gray to The Economics of Bandwidth.

Who said that (first)?

Naturally, a Google search produces hundreds of citations about the bandwidth of station wagons; I have included a few interesting ones I found while writing this. But as with many familiar sayings, it seems that the station wagon analogy has evolved over 25 or more years, and its exact origins are now hard to pin down. A BBC UK article highlights some of this uncertainty. An article at SysAdmin humor cites Dennis Ritchie as a possible source, but then admits ignorance. The Wikipedia article (today) on Sneakernets does attribute both the current and an earlier version of the saying to Tanenbaum, but then descends into a stew of "alleged" references and possibilities.

The original version of this quotation came much earlier; the very first problem in Tanenbaum's 1981 textbook Computer Networks asks the student to calculate the throughput of a St. Bernard carrying floppy disks (which are said to hold 250 kilobytes of data). The first USENET citation is July 16, 1985, and it was widely considered a chestnut already, possibly dating from the 1970s. Other alleged speakers included Tom Reidel, Warren Jackson, or Bob Sutterfield. The station wagon and mag tapes were the canonical version, but variants using trucks or Boeing 747s and later storage technologies such as CD-ROMs would frequently appear.

—Wikipedia article on Sneakernet [July 12, 2007]

I will start a thread in my Who said that? discussion forum, just in case anyone wants to add something more concrete!

Next ...

In my next post, I will continue my review of Alberto Savoia's 2001 paper [**], and explain the relevance of today's digression.

[** Warning: even though it's just 6 pages, it's a 2.5Mb file, so wait until you're on a fast connection].

Tags: William Dougherty, edgeblog, performance, bandwidth, throughput, latency, round-trip time, Stuart Cheshire, Web performance, Web application, download time, Alberto Savoia, Steve Harris, DSO Guy, Fedex, Royal Pingdom, Jonathan Schwartz, sailboat, Hong Kong, station wagon, Andrew Tanenbaum, Jeff Atwood, Coding Horror, Jim Gray, sneakernets, Performance Matters

Web Performance Matters

jQuery Library Performance Alert

jQuery libraries moved to Google AJAX Libraries API

code.jquery.com Redirected to Google Ajax APIs

Where Performance Meets Availability

Response Time Standards for Web Sites and Web Applications

Availability

Why Technorati is Not Usable

Availability

Performance

Clarity

Utility

Trying Out Blog Widgets and Tools

The bottom line

Human Factors and Blog Design

Thirteen blog clichés

1. The Useless Calendar Widget

2. Random Images Arbitrarily Inserted In Text

3. No Information on the Author

4. Excess Flair

5. The Giant Blogroll

6. The Nebulous Tag Cloud

7. Excessive Advertisements

8. This Ain't Your Diary

9. Sorry I Haven't Written in a While

10. Blogging About Blogging

11. Mindless Link Propagation

12. Top (n) Lists

13. No Comments Allowed

If I Had A Hammer ...

Scalability is Not Optional

Other posts about scalability

Proofread before publishing

Huh?

Managing for Business Effectiveness

Drucker on Effectiveness vs. Efficiency

Three key questions

A principled foundation

Asynchronous Architectures [4]

Introduction

3:45: What is Scalability? [slide 2]

13.1 The Parallelism Principle: Exploit parallel processing

13.2 Scalability and speed up

8:15: Scalability for Real Systems [slide 3]

8:45: But ... [slide 4]

10:05: Principles for Scalable Service Design [slide 13]

The Mythical Man-Month

26:00: Scalability Through Smart System Engineering [slide 12]

30:40: CAP Conjecture [slide 8]

32:00: A Clash of Cultures [slide 5]

34:05: ACID vs BASE [unnumbered]

35:50: Why the Divide? [Slide 7]

39:30: Consistency and Availability [slide 9]

44:00: Partition-Tolerance and Availability [slide 10]

45:40: Techniques [slide 11]

50:30: What does this mean for the data architecture?

Asynchronous Architectures [3]

Dan Pritchett's Design Principle

Latency hurts customer service

Latency tolerance

Data partitioning

ACID or BASE?

References: business-driven or event-driven architectures

Next: the CAP theorem

Asynchronous Architectures [2]

The Fallacies of Distributed Computing

Starting from the wrong place

Starting from the right place

Asynchronous architectures may reflect manual systems

Business process design

Asynchronous Architectures [1]

Bell's Law of Waiting

Asynchronous architectures are more scalable

Design applications that don’t wait

Asynchronous vs. Synchronous Consumers

Bell's Law

Five Scalability Principles

Five Scalability Principles

Learning the hard way

1. Don’t think synchronously