I was going to write about performance and availability today, but this was not the post I had in mind. Technorati sidetracked me. So I'm going to write about Usability instead. Because Technorati provides a good counter-example -- how not to build a usable Web application that satisfies and retains customers.

In Web Usability: A Simple Framework, I described a way to think about Web site or Web application usability.

In a second post, The Dimensions of Usability, I presented the graphic shown here, and discussed the four dimensions in a bit more detail.

These four dimensions are not alternative functional goals, to be weighed against one another and prioritized. Web application effectiveness is a four-step challenge:

To satisfy customers, a Web site must fulfill four distinct needs:

• Availability: A site that's unreachable, for any reason, is useless.
• Responsiveness: Having reached the site, pages that download slowly are likely to drive customers to try an alternate site.
• Clarity: If the site is sufficiently responsive to keep the customer's attention, other design qualities come into play. It must be simple and natural to use – easy to learn, predictable, and consistent.
• Utility: Last comes utility -- does the site actually deliver the information or service the customer was looking for in the first place?

--Web Usability: A Simple Framework, October 17, 2005

As in a quiz show, to win the grand prize -- satisfied customers -- you have to get it right at every stage. Fail at any one and you will lose customers. Fail consistently at any one, and you will be out of business.

As I experienced their service today, Technorati seemed to be failing on all four fronts.

Availability

First I noticed that my browser was replacing the thumbnail portrait that usually appears (near the bottom of my sidebar) under technorati links with alternative text. Next I tried Technorati's link to Blogs that link here and, eventually, was rewarded with:

It's not that I desperately need to see my own picture there at all times, or that I think my readers are dying to see the inbound links (or blog reactions, as Technorati calls them). I know that widget in my sidebar has marginal utility -- a few people may use it occasionally. That's why I put it near the bottom, where it doesn't interfere with anyone's ability to browse the site. If it works, it does no harm. On the other hand, if it's broken, it becomes a distinct liability. Broken links lower the quality of the whole site..

In this case, the connection reset error means that the Web server accepted the request, but then took so long to respond that the browser timed out. While this may not qualify as a broken link, it had the same effect: the requested page was unavailable.

Upon checking back a few hours later, the sidebar link was working. Again, not a surprise. Intermittent outages like this have been characteristic of Technorati for a long time -- see my post on Taming the Technorati Monster.

Performance

I'm not going to dwell on this, because if you've tried to find things on Technorati lately, you already know how the service performs. For me, it typically ranges from slow to glacial, for a search engine. Maybe it's just my particular interests -- Web Performance and Application Responsiveness are not especially hot topics. Perhaps other people who are interested in more popular topics are scoring cache hits and are actually getting good Web performance. That would be ironic!

Clarity

Returning to my problems with the blog reactions widget, normally I would let this incident pass without comment. But today I actually noticed the problem while I was doing a blog search about problems with Technorati's tag indexing and search functions, because my blog seems to have fallen off their radar lately.

In the past, tags used in a post would be indexed and returned in a search within the hour, often within minutes. Today, Technorati's search function was sure I had not published anything for the last 27 days. But when I navigated manually to their page for my blog, they displayed excerpts of all my more recent posts.

Some more digging revealed that even though Technorati's site was up, and even though I could use it to navigate manually to a page listing blog reactions, the sidebar link to that same information, which Technorati's widget was generating, did not work. In any Web site or application, these kinds of internal inconsistencies are hugely frustrating. They make me doubt the accuracy and completeness of any information the application returns.

And not surprisingly, searching for help on Technorati itself does nothing to reassure me that they will be fixing these problems anytime soon. Quite the contrary, it confirms their problems, as in this amusing response:

(Is Technorati beatin' 'round the bush here? Can a search engine actually lie about what it really knows? :)

Utility

Google's Blog Search feature, however, did know something. It pointed me to this sensible post by ChristineMM:

Trying Out Blog Widgets and Tools

I would like a search box on my blog that lets my readers (and me) search for content from within the pages of my blog. The Google one that I used to use was not working right, I’d search for a word that was right in front of me or even in a blog post title and it would say there were no matches, so I dumped that function.

I then for a long time used a Technorati box. At some point I realized it was not working well either. Again I’d search for a keyword that was in a blog post title and it would say there were no matches. So this week I deleted it from my sidebar. What is the point of having a blog reader search for a topic on my blog, be told I never blogged on it, when in reality, I actually did?

...

One last thing I’ll mention is that I get a ton, and I mean a ton of blog readers through Google primarily and also some other Internet search engines. I feel that my regular use of Technorati tags helps my blog posts be found by Google and the other search engines. This drives traffic to my blog. So if you want to drive traffic to your blog, use Technorati tags in every blog post of substance.

--The Thinking Mother, September 23, 2007

Well said, Christine! If an application does not deliver the service you need, it's useless. I dropped Technorati's search box some time ago, for similar reasons. The integrated Squarespace search box is 1000 times more useful for searching the blog, and Google has the Web covered far more effectively than Technorati, in my opinion.

The bottom line

I think Christine may have homed in on the essence of the matter, sad though it may be. Unless Technorati can recover its original sense of purpose and fix its technical problems, it's not going to survive as an independent, useful, service. Perhaps its most significant contribution will be its promotion of a standard tagging format that is easily recognized and reused by other search engines.

So I'm not giving up on my Technorati tags yet, but I'm not counting on getting much value from their blog indexing or searching tools either. I've already removed their blog search and tag cloud functions from my sidebar, and their blog reactions widget is now on probation. Any more problems and it will be the next to go.

Tags: Technorati, tagging, blog reactions, problems, usability, availability, consistency, clarity, utility, Christinemm, Thinking Mother, Web performance, Performance Matters

The best products are designed with Human Factors in mind. That's why Web design and usability is a frequent topic of my Web Performance Matters blog.

Jeff Atwood's blog -- Coding Horror -- focuses on programming and human factors. And according to a recent interview with Jeff on the site Daily Blog Tips, "the blog is attracting over 500,000 unique visitors every month, and it also counts 60,000 RSS readers, meaning that Jeff probably knows what he is talking about".

The Coding Horror logo was originally created to mark examples of dangerous code in the programming classic Code Complete by Steve McConnell, which Jeff rates as his "all-time favorite programming book."

I have recommended Code Complete myself. Jeff's favorite books are on my shelf too. So I respect his judgment and recommend his blog, which I have added to the blogroll on Web Performance Matters.

Thirteen blog clichés

Jeff recently published Thirteen Blog Clichés, a post summarizing his "opinions about what makes blogs work well, and what makes blogs sometimes not work so well." These are presented as a list of common mistakes to avoid (or anti-patterns). If you have a blog, or are designing one, you've probably read similar articles before. Even so, Jeff's checklist is worth a look. All such lists tend to contain a core set of common guidelines to follow and/or pitfalls to avoid, but some of Jeff's opinions step outside the conventional wisdom.

Because I maintain two blogs -- Web Performance Matters and UpRight Matters -- I decided to rate both blogs against Jeff's criteria. Here are edited versions of his recommendations, and my responses. To read Jeff's full discussions of each guideline, see the original. And for the full story, see the many responses posted by Jeff's readers in the comments section of his blog.

1. The Useless Calendar Widget

I can't think of a single time I have ever found the blog calendar widget helpful. My computer already has a calendar function, so it's not like I need another calendar displayed in my web browser.

Every post carries an obvious datestamp, so I can easily discern when it was published. But knowing whether someone posted an entry on the third Tuesday of the month? Utterly useless.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree! Someone reading a blog like Daily Kos, that publishes daily about politics or current affairs, might find a calendar useful. But a calendar isn't appropriate for our content, so we've never thought of including one. Even if we had, the Squarespace publishing platform we use (see bottom of sidebar) doesn't offer such a blog calendar widget -- another sign that it's not in great demand.

2. Random Images Arbitrarily Inserted In Text

One of the cardinal rules of web writing is to avoid large blocks of text. There are plenty of excellent web writing guides that exhort you to break up your text, using bullets, numbered lists, quotes, paragraph breaks, images -- anything to avoid creating an intimidating wall of dense, impenetrable text.

But like all good advice, (this) can be taken too far. For example, when you find yourself inserting random pictures into your writing for the sole purpose of breaking up the text. As the old adage goes, a picture is worth a thousand words. But you should no more insert a random image into your writing than you would insert a thousand random words into your writing.

Images are not glorified paragraph breaks. Images should contribute to the content and meaning of the article in a substantive way. And if they don't, they should be cut. Mercilessly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I know I am sometimes guilty of writing long posts. But I won't write a thousand words unless I have something worthwhile (I hope :-) to explain, and I try to keep all my posts interesting by breaking up the text using headings or images. And I promise that we will never include an image that bears no relationship to the subject matter!

3. No Information on the Author

Every time a reader encounters a blog with no name in the byline, no background on the author, and no simple way to click through to find out anything about the author, it devalues not only the author's writing, but the credibility of blogging in general.

Maintaining a blog of any kind takes quite a bit of effort. It's irrational to expend that kind of effort without putting your name on it so you can benefit from it. And so we can too. It's a win-win scenario for you, Mr. Anonymous.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree! That's why we provide brief introductions and longer author pages.

4. Excess Flair

Blogs work because they're simple. When we clutter up our blogs with a zillion widgets, features, and add-ons, we're destroying an essential part of what makes blogs worthwhile. Examples include "crazy" JavaScript image loading techniques, annoying pop-up image previews of links, and pictures of the last 10 visitors to your blog.

Before adding any new "feature" to your blog, consider whether its value outweighs the additional complexity it introduces.

--Jeff Atwood, Thirteen Blog Clichés [edited]

This recommendation can be controversial -- see the comments on Jeff's original post on this topic. But I agree with Jeff, and I do try to reduce the clutter whenever possible. See my decision on item 6 below, for example.

5. The Giant Blogroll

Citing your references and influences is a great and necessary thing, but obsessively listing every single blog you read is just noise. If you're really reading this many blogs, you should be linking to them organically in your blog posts, in a sort of natural quid pro quo. Wearing a giant blogroll on your sleeve is an empty gesture that feels artificial and insincere.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Agreed! On Web Performance Matters, I aim to keep my blogroll focused, and group the links into categories. We have not added a blogroll on UpRight Matters yet, but we plan to adopt the same approach.

6. The Nebulous Tag Cloud

Tagging content easily beats organizing everything into hierarchical folders, and tag categories on blogs are moderately useful, particularly for bloggers who tend to bounce around among many different topics. What I've never found useful, however, is the stereotypical tag cloud visualization, where the size of the tag word varies with its frequency.

The perception is that tag cloud visualizations are cool, like badges of honor for the tagging club. The reality is that tag cloud visualizations are chaotic, noisy, and unusable. Keep the tagging, lose the cloud. A simple sorted list of tags, along with the number of posts associated with each tag, is much more effective.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Content tagging and indexing is a complex subject, and one I have given much thought while developing our blogs. [I even read Everything is Miscellaneous, and started to write a post about it until I realized that I was just adding to the echo-chamber on that topic. See item 11 below.]

I believe that tagging with keywords has value, but the resulting folksonomy is most useful as a supplement to, not a replacement for, a carefully designed and consistently applied classification scheme or information architecture. Therefore we will continue to index our content using both methods.

However, despite investing a lot of time implementing a Technorati tag cloud for Web Performance Matters, which has been sitting in my sidebar for 4 months, I have come to the same conclusion as Jeff -- it takes up space without adding any value. So I've now removed it.

I see this as an example of the dynamic nature of blogging. It's agile publishing: you don't have to get everything right the first time. You can create something, try it out for a while, refine it, or remove it altogether. In this vein, revising your blog's layout is greatly simplified if your publishing platform is CSS-based, like Squarespace.

7. Excessive Advertisements

Advertising is a fact of life, but your blog is not Times Square. Does every square inch of whitespace have to be filled with paid links, Google AdSense, and ad banners?

Here's a related article on blog usability that's a perfect -- even ironic -- example of how you can hurt your usability with excessive, obnoxious advertising. It's everywhere.

It is almost never in the reader's interest to see advertisements, so tread very lightly, and be respectful of your audience. If you take the time to advertise responsibly, you may find that readers appreciate you for it.

Well, probably not, but it can't hurt to try.

--Jeff Atwood, Thirteen Blog Clichés [edited]

We do have a few ads. I try to organize them tastefully, so that they don't interfere with the content.

8. This Ain't Your Diary

Let's be perfectly clear: readers aren't coming to your blog to read about you. They're coming to find out what it can do for them.

That said, blogs are a place for writers to find an interested audience, and a place for readers to find a helpful peer and a unique voice. It's OK to be yourself; at some level, it is a cult of personality: people are reading not only because your content is useful to them, but because they like you.

It's normal to inject a regular dose of yourself into the conversation. But like Tabasco sauce and other powerful seasonings, a little YOU goes a long way. A really long way. Write accordingly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Agreed! I won't be writing about my experiences remodeling my house, unless I see a connection worth exploring.

9. Sorry I Haven't Written in a While

If you haven't posted anything new to your blog in a while, don't waste our time with apologies. Just write! The best apology is new and improved content. Maybe with a wee bit more consistency this time, though:

• Pick a schedule you can live with, and stick to it
• Don't produce substandard posts, just to keep to a schedule
• Talent is far less important than enthusiasm

And the best way to demonstrate your enthusiasm -- and to improve -- is to get out there and write. Regularly.

--Jeff Atwood, Thirteen Blog Clichés [edited]

My objectives for Web Performance Matters are much the same as when I started the blog two years ago -- to contribute an organizing framework and a regular supply of ideas. I have to admit, I've had a few long gaps in my writing. I've also apologized and promised to do better! But after reading Jeff's advice, I'm in a bind. Should I apologize for apologizing? I guess not. I'll just keep writing.

10. Blogging About Blogging

I find meta-blogging -- blogging about blogging -- incredibly boring. I said as much in a recent interview on a site that's all about blogging (hence the title, Daily Blog Tips). If you accept the premise that most of your readers are not bloggers, then it's highly likely they won't be amused, entertained, or informed by a continual stream of blog entries on the art of blogging.

Meta-blogging is like masturbating. Everyone does it, and there's nothing wrong with it. But writers who regularly get out a little to explore other topics will be healthier, happier, and ultimately more interesting to be around -- regardless of audience.

--Jeff Atwood, Thirteen Blog Clichés [edited]

Of course, Jeff's post and this one are about blogging. But the reason we care enough to research and write about these ideas comes back to the Human Factors dimension. As we work to improve our ability to serve and communicate more clearly, we want to share what we learn to help you connect with your own readers and communities.

Cynthia writes:

11. Mindless Link Propagation

One of the most pernicious problems in blogging is the echo chamber effect. Most blog entries merely regurgitate what other people have said or add vapid commentary on top of news articles and press releases. Only the tiniest fraction of blog entries are original content, and only a tiny fraction of that fraction is worth your time.

If everyone knows about it, what value does that information have? My advice here is almost contrarian: if everyone else is talking about it, that means you should avoid talking about it. Switch things up. Seek out uncommon sites with unique information. If all you can find to talk about is what's already popular, you're not trying hard enough. Form your own opinion. Do your own research. Go out of your way to blaze a new trail and create something we haven't already seen hundreds of times before.

--Jeff Atwood, Thirteen Blog Clichés [edited]

That is so true! This issue is exactly what stopped me from completing my review of David Weinberger's book, Everything is Miscellaneous, even though I had read the book and dozens of pages of reviews by others. But then I found myself summarizing other readers' feedback, which amounted to a folksonomy about the topic of folksonomies -- that is, a meta-folksonomy. I began to wonder whether, if I were to review other similar discussions, and add some Technorati tags to my review, would I then be contributing to a meta-meta-folksonomy?!

People criticize Web 2.0, and the blogosphere in general, claiming that it's just a giant echo-chamber in which uninformed opinion is amplified, and true expertise is drowned out by uneducated bleating, like the sheep in Animal Farm. While that criticism may sometimes be true, it is not the whole story, and it does not do justice to the educational power of the Web.

In this case, I concluded that the world did not really need me to summarize what everyone else was saying about David Weinberger's opinions about tagging, folksonomies, and the wisdom of crowds. In fact, to write such a post would just fuel the critics' argument (and by the way, it was turning into a very long post). So I went back to writing about Web performance!

12. Top (n) Lists

Yes, exactly like this one.

Lists are a great convention. They make sense, people understand them, and they're a logical way to structure your writing. But don't let lists become a crutch. I'm always taken aback when I see the "most popular" posts on a blog dominated by Top (n) Lists. Shortcuts are only meaningful if you know what it is, exactly, you're cutting.

If you find that the Top (n) List convention is a go-to tool in your writing toolkit, consider re-balancing your writing portfolio with longer, more in-depth pieces as well. Not everything should be a sprint; throw a few small marathons in there somewhere to complement your short distance skills.

--Jeff Atwood, Thirteen Blog Clichés [edited]

I agree that it's good to aim for a balance of short and longer posts. Having written technical articles for years before blogs existed, I'm actually more likely to write long essays -- see my comment on item 2 above. So to balance those marathon posts, I've found that deliberately trying to compose shorter posts whose structure is a list of guidelines or principles helps me keep that balance. My series on Performance Wisdom contains several examples of this approach.

13. No Comments Allowed

A blog without comments is not a blog. Yes, there are exceptions for massively popular blogs where comments don't scale. But until that applies, the value of the two-way conversation far outweighs any minor inconvenience on your part. It's an open secret in the blogging community that the comments are often better than the original blog entry itself. Would you browse Amazon without the user reviews?

Don't be afraid of comments. Embrace them. Moderate them. The community will respect you for it, and your blog will be better for it as well.

--Jeff Atwood, Thirteen Blog Clichés [edited]

We want comments! One of the primary reasons for blogging is to have conversations about the things that matter. It's not about us. Well, most of the time, anyway -- so we won't apologize if our writing occasionally lapses into introspection or vanity.

Note: This is cross-posted on Web Performance Matters and UpRight Matters.

Tags: human factors, Jeff Atwood, Coding Horror, blog clichés, calendar widget, random images, social bookmarks, blogroll, tagging, folksonomy, Web 2.0, tag cloud, blog advertising, meta-blogging, echo chamber, David Weinberger, Everything is Miscellaneous, Animal Farm, blog comments, blog design, Web Performance Matters, UpRight Matters

If I had a hammer
I'd hammer in the morning
I'd hammer in the evening
All over this land
I'd hammer out danger
I'd hammer out a warning
I'd hammer out love between my brothers and my sisters
All over this land

--Pete Seeger and Lee Hays, 1949 [Wikipedia]

In May 2007, after I wrote about Controlling What You Can't Measure, I had a conversation with Ben Simo (see the comments) about metrics and tools, during which I wrote:

Hammers and chisels can be very dangerous, but carpenters use them every day, and we don't brand them as "bad tools" just because some people don't know how to use them properly. Human nature being what it is, there will always be some incompetent fools who try to use a hammer and chisel to drive in a screw or open a bottle of beer, just because they have those tools handy.

A few days later, while reading a series of articles on Performance Engineering written by Scott Barber, I noticed the following quotation:

All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.

--IBM maintenance manual, 1925 [emphasis added]

This priceless piece of advice is quoted by Scott in part 9 of his 14-part series. At the time I made a note to write something about this, but after that it just sat in the "ideas for blog posts" folder for the next 3 months.

Until today, when I happened across the Ultimate Geeks Multi-Tool Hammer. Now, if I had this hammer, it turns out that I actually could use it to drive in a screw or open a bottle of beer, without being branded as an incompetent fool!

Tags: Scott Barber, Ben Simo, multi-tool, hammer, Performance Matters

My recent post, Asynchronous Architectures [4], summarized a presentation by Werner Vogels at the 2007 QCON conference in London.

A subsequent post by Kent Langley in his new ProductionScale blog -- entitled Getting Rid of the Relational Database -- supports the arguments advanced by Vogels.

Describing the relational database model as "the proverbial ball and chain in the relationship between scalable applications and the underlying infrastructure," Kent writes:

The quest for seamless linear growth for technology applications is being hindered by the “elephant database.”

What would Amazon do? In a recent talk2 at QCON London Werner Vogel, the CTO of Amazon.com clearly noted that the relational database model is a essentially outdated for the needs of modern applications as a primary data storage medium. In other words, it is simply to slow and cumbersome.

Additionally, Mr. Vogel makes a critical point that in many, many cases relational databases are simply not necessary. Simple key/value pairs (hashes) are all you need.

--Joseph Kent Langley, Getting Rid of the Relational Database, August 11, 2007

Kent goes on to describe why he believes "you should break out of a one-size-fits-all way of thinking when it comes to databases, data storage, and scalable systems. Vertical scaling by throwing hardware at it is no longer sufficient for modern web scale applications".

Kent also points to Data 2.0: How the Web disrupts our relational database world, which he admits he has not read. Maybe if he had read the article he might have omitted this link.

Although the author of that article, Nitin Borwankar, supports Vogel's general conclusions, his style is to make sweeping pronouncements. He advances no technical arguments that lead up to his conclusion that "The days of Data 1.0 are past. The days of Data 2.0 are dawning, and it promises to be very disruptive for mainstream database architectures on the Web".

Other posts about scalability

I recommend Kent's new blog, and I'm adding it to my blogroll. It looks as if it will contain regular discussions of performance topics. For example, since launching the blog in early August, Kent has already written about:

• Scalable LAMP: Caching
• Varnish - A Web Accelerator [a fast reverse proxy caching system]
• The Power of mod_deflate [about content compression]
• Scalability and Performance: A Few Resources
• Testing with WBox [about scalability testing]

Proofread before publishing

I do have one small complaint. As a blogger, I know the feeling of writing down my thoughts and wanting to get them published -- right now! The short publishing cycle is one of the attractions of a blog. But if Kent could just curb his enthusiasm for long enough to proofread his posts once carefully before hitting that Publish button, his thoughts would be a lot easier to follow:

Huh?

• Afterwards, I will follow up with a brief analysis or executive summary if you will of this infobit might mean for businesses. Needs commas around "if you will," and "of" should be "of what".
• By way if example using the techniques of bond arbitrage Stonebraker notes quite earnestly that it is a “latency arms race.” Needs commas around "using ... arbitrage," and "if" should be "of".
• So, is this inconclusive evidence of the pending death of the Relational Database? Of course not. "Inconclusive" should be "conclusive".
• But, it is trend spotting in that people are again noticing that there are other ways and that those other ways just might quite faster with modern applications. "might quite" should be "might be".

--Joseph Kent Langley, Getting Rid of the Relational Database, August 11, 2007

Although I can infer what Kent is trying to say here, these glitches spoil the overall effect by forcing me to re-read his sentences to get the point. In elementary school, I learned the old English proverb: Don't spoil the ship for a ha'pworth of tar. I believe bloggers should consider readability to be as important as their message, if they want to build a faithful following.

Tags: scalability, Kent Langley, ProductionScale, Werner Vogels, QCON, relational database, Web applications, Performance Matters

There is surely nothing quite so useless as doing with great efficiency what should not be done at all

-- Peter Drucker, 1963

Peter Drucker is often called "the father of modern management". Many books and Web sites are devoted to his insights, some of which I have written about previously.

This post highlights his incisive observation about the difference between effectiveness and efficiency. I have always found it to be especially memorable, and quoted it (twice) when discussing priorities and choices in my book about software performance. Unfortunately I got the source wrong, but thanks to Google I can now correct my mistake.

It appeared in Managing for Business Effectiveness, an article in the May/June 1963 edition of Harvard Business Review ("HBR"). You can also find it in a February 2006 HBR article -- What Executives Should Remember -- a collection of excerpts drawn from HBR articles by Drucker published between 1963 and 2004.

Because Drucker's remarks are equally relevant to technical performance management and business leadership, I am cross-posting this here on Web Performance Matters, and on our new UpRight Matters blog.

Three key questions

In his 1963 essay, Drucker states that there is no magic formula, checklist, or procedure that will substitute for the hard, demanding, risk-taking work of management. But he claims that "we know how to organize the job of managing for economic effectiveness and how to do it with both direction and results. The answers to the [following] three key questions ... are known, and have been known for such a long time that they should not surprise anyone."

1. What is the manager's job? It is to direct the resources and efforts of the business toward opportunities for economically significant results. This sounds trite—and it is. But every analysis of actual allocation of resources and efforts in business that I have ever seen or made showed clearly that the bulk of time, work, attention, and money first goes to "problems" rather than to opportunities, and, secondly, to areas where even extraordinarily successful performance will have minimum impact on results.

2. What is the major problem? It is fundamentally the confusion between effectiveness and efficiency that stands between doing the right things and doing things right. There is surely nothing quite so useless as doing with great efficiency what should not be done at all. Yet our tools—especially our accounting concepts and data—all focus on efficiency. What we need is (1) a way to identify the areas of effectiveness (of possible significant results), and (2) a method for concentrating on them.

3. What is the principle? That, too, is well-known—at least as a general proposition. Business enterprise is not a phenomenon of nature but one of society. In a social situation, however, events are not distributed according to the "normal distribution" of a natural universe (that is, they are not distributed according to the U-shaped Gaussian curve). In a social situation a very small number of events—10 percent to 20 percent at most—account for 90 percent of all results, whereas the great majority of events account for 10 percent or less of the results.

-- Peter Drucker, Managing for Business Effectiveness, Harvard Business Review, May/June 1963

A principled foundation

Although Drucker writes about management effectiveness in the context of business performance, specialists in software or systems performance must ask the same questions and apply the same principles. In my book on performance management [Amazon], I described these principles as follows:

• The Centering Principle: Focus on the most performance-critical components.
• The Efficiency Principle: Maximize the ratio of useful work to overhead.
• The Pareto Principle: Prioritize the 20% of the problem that will return 80% of the benefits.

Drucker concludes that the most crucial requirement for effective management is having ...

... the courage to go through with logical decisions -- despite all pleas to give this or that product another chance, and despite all such specious alibis as the accountant's "it absorbs overhead" or the sales manager's "we need a full product line."

This is one small example of the characteristic I find so appealing in Drucker's writing. His advice starts from an assumption that there are relevant principles, and that you can make decisions by reasoning logically from those foundations. As a mathematician, this way of looking at the world appeals to my sense of order and logic, rather than presenting me with a collection of unsupported assertions and beliefs. The HBR introduction to its review, What Executives Should Remember, sums up Drucker's appeal as follows:

Executives had come to think they knew how to run companies, and Drucker took it upon himself to poke holes in their beliefs, lest organizations become stale. But he did so in a sympathetic way. He assumed that his readers were intelligent, rational, hardworking people of goodwill. If their organizations struggled, he believed it was usually because of outdated ideas, a narrow conception of a problem, or internal misunderstandings. His insights were ... practical idea-based essays for executives, and his clear-eyed humanistic writing enhanced the magazine time and again. He helped us all to think broadly and deeply.

-- What Executives Should Remember, Harvard Business Review, February 2006

This is why so many people have enjoyed, and like Harry Joiner, continue to enjoy daily, the wisdom of Peter Drucker.

Tags: management, management principles, management wisdom, Peter Drucker, effectiveness, efficiency, Pareto Principle, 80-20 rule, Harry Joiner, UpRight Matters

This is the fourth in a series of posts presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment.

In a QCon conference presentation on Availability and Consistency or how the CAP theorem ruins it all, Werner Vogels, Amazon CTO, examines the tension between availability & consistency in large-scale distributed systems, and presents a model for reasoning about the trade-offs between different solutions.

I recommend you find time to watch the entire 52-minute video. The flash streaming technology that InfoQ uses is subject to buffering hiccups, and you may have to restart it a few times. So in case you want to jump to a specific section, I've assembled copies of Werner's slides, with short timestamped notes on the content of each section. Werner did not present his slides in their numbered order, so in my notes I identify slides using the numbers printed on them, not their presentation order.

Introduction

0:50: CTO's must match business with technology. Most really big IT shops must push the edge of what commercial technology can do. Technology has a very long adoption cycle -- it takes about 10 to 15 years for new technology to mature and be effective. For leading companies like Amazon, that's too slow, the scalability challenges are so great that they demand advanced solutions. So shops are forced (in effect) to do their own research, take advanced steps, just to succeed in a competitive marketplace.

2:15: Werner noted that his viewpoint disagreed strongly with that of Cameron Purdy, CEO of Tangosol, who was an advocate of database technology.

3:00: He introduced Eric Brewer's CAP theorem -- more later. [See the end of my previous post in this series, Asynchronous Architectures [3]. The CAP theorem was first propounded in a 1998 presentation -- Lessons from Internet Services: ACID vs. BASE -- by Dr. Eric Brewer of Inktomi, now a professor at UC Berkeley].

3:45: What is Scalability? [slide 2]

3:45: The meat of Werner's QCon presentation really begins here. Proportional is the key word in these definitions. Adding resources should deliver increased capacity proportional to the added resources. Or if the intent was to deliver better performance, the gains should be proportional to the added resources. Performance here is not just about response, it could mean transfering more data or larger datasets.

4:40: Another reason for needing scalability is to achieve fault-tolerance. Adding resources to achieve redundancy should not hurt your performance. Traditional technologies (like databases) won't give you this kind of scalability, because overheads increase as you scale up. These are subjects I discussed at length when explaining The Parallelism Principle in High-Performence Client/Server:

13.1 The Parallelism Principle: Exploit parallel processing

Processing related pieces of work in parallel typically introduces additional synchronization overheads, and often introduces contention among the parallel streams. Use parallelism to overcome bottlenecks, provided the processing speedup offsets the additional costs introduced.

13.2 Scalability and speed up

Scalability refers to the capacity of the system to perform more total work in the same elapsed time, when its processing power is increased.

Speed up refers to the capacity of the system to perform a particular task in a shorter time, when its processing power is increased.

In a system with linear scalability and speed up, any increase in processing power generates a proportional improvement in throughput or response time.

--High-Performance Client/Server, Chapter 13, pp383-385.

8:15: Scalability for Real Systems [slide 3]

7:00: Slide 3 is the conclusion of a cost discussion that begins before the slide is shown. The biggest threat to availability is bugs, which are a cost factor introduced by humans. So operating costs must not grow as you scale up.

8:45: But ... [slide 4]

8:30: Traditional technologies, databases, two-phase commit may work for 2-4 nodes, but they will not scale to 100's (let alone 10.000) nodes. You may not have 10,000 nodes like Amazon, but you will run into these scalability challenges at 50-100 nodes.

10:05: Principles for Scalable Service Design [slide 13]

10:05: Guidelines for services design at Amazon -- a checklist of lessons learned through hard experience:

• 10:05: Decentralize. Any algorithm that requires agreement will eventually become a bottleneck. Two-phase commit is an in effect an unavailability algorithm, it is guaranteed to fail as you scale up the number of participating services.
• 10:50: Asynchrony. Make progress under all cicumstances, even if the world is burning around you. Even if fulfillment services are burning down, you want people to be able to place orders. So work locally, don't worry about the rest of the system.
• 11:40: Autonomy: Each node should be able to make decisions based only on local state. If you need to reach agreement based on global conditions at high load, you are lost. Nodes may be failing, coming up, going down all the time. Probabilistic techniques work well in these circumstances.
• 12:35: Controlled concurrency. Reduce concurrency as much as possible, so that you do not need to use locking.
• 13:15: Controlled parallelism. Control traffic going to each node using careful load balancing; nodes must have spare CPU and I/O capacity so that they can do other tasks (like load re-balancing) in the background.

• 14:10: Symmetry. Things work really well if all nodes do exactly the same thing. It is easy to add more nodes if nodes do not have to be identified as a directory node, a data-storage node, etc. Ideally, you just install the software and run it, and it responds to any client request and does whatever task is needed, or maybe forwards the request if necessary. This is the logical way to address a requirement that I first documented in a 1993 paper, and later included in Chapter 16, Architecture for High Performance, of my book:

For a large organization, moving to an enterprise client/server system represents a major shift from monolithic systems with fixed distribution to dynamic, heterogeneous, pervasively networked environments. The next generation of systems will be an order of magnitude more dynamic--always running, always changing--with thousands of machines in huge networks.

In such an environment, content components (service providers like DBMSs) and service consumers (e.g. GUIs) must be continually added and removed.

The key to doing this is the middle tier, the hub of the three tier architecture. In the first place, this central layer acts in a connecting role to let individual clients access multiple content servers, and (of course) servers support multiple clients. A separate central tier can also:

• Provide a set of services that can be dynamically invoked by both the consumer and content layers
• Allow new services to be added without major reconfiguration
• Allow services to be removed dynamically without affecting any participant not using those services
• Allow one service provider to be replaced by another

These are all vital characteristics in a distributed computing environment.

--High-Performance Client/Server, Chapter 16, pp514-515.

15:10: Algorithms that force you to obtain agreement will become a bottleneck. So avoid using two-phase commit, maybe by denormalization to make sure your transaction always runs within a single node. Or split your task into multi-transaction workflows. You have to take an end-to-end look at the business transaction and decide. [I have always advocated this approach -- see the conclusions of the second post in this series].

17:40: You can reuse some of those principles in building teams. Small teams are best, so that each team is responsible for a well-understood piece. Team effectiveness is just as important as architectural consistency.

19:00: We call this the two pizza rule -- If you can't feed a team with two pizzas, it's too big. OK, hungry just-out-of-college students do eat more, but they work harder too! As soon as you need more than 8 people, it's hard to understand what everyone is doing. Bigger teams, of 12 or more, must have meetings, and must spend a much larger percentage of their time communicating. This discussion harks back to the famous observation by Fred Brooks:

The Mythical Man-Month

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them...

In tasks that can be partitioned but which require communication among the subtasks, the effort of communication must be added to the amount of work to be done...

The added burden of communication is made up of two parts, training and intercommunication...

If each part of the task must be separately coordinated with each other part, the effort increases as n(n-1)/2. Three workers require three times as much pairwise intercommunication as two, four require six times as much as two. If, moreover, there need to be conferences among three, four, etc., workers to resolve things jointly, matters get worse yet. The added effort of communicating may fully counteract the division of the original task.

--The Mythical Man-Month, Frederick P. Brooks, 20th Anniversary Edition pp17-18, Addison Wesley, 1995.

21:20: At Amazon, not all the services (1000 or more) support Web interactions at Amazon.com. Many are in the back-end systems such as supply-chain, fulfillment, enterprise services, handling feeds from 3rd-party suppliers, item management, recommendations, personalization.

22:15: Example of statistically improbable phrases (SIPs), an interesting digression about text analysis being implemented as yet another service. (Too long. The details are a plug for Amazon, but take time away from the main thread of the presentation).

24:30: Conclusion of the SIP discussion -- you need dependency management, and contracts. Servers can give an SLA based on workload conditions, that clients must honour. Automatic dependency discovery -- Amazon has home-grown tools that can show where dependencies exist, and the effects of failures in a network of nodes.

26:00: Scalability Through Smart System Engineering [slide 12]

26:00: Use scalable primitives. For example, RPC is not scalable. Don't conceal heterogeneity. We can pretend that systems don't fail, but in practice they do. That's the problem with RPC, it pretends to be a procedure call but it isn't, so transparency does not really work, failures do happen, performance differences do exist. So don't conceal these differences.

28:00: Configuration management. If you have 1000 services, configuration becomes really important. If your applications involve strong consistency properties, they can create problems when people leave your team.

29:00: Repair and recovery. Check out the work on Recovery Oriented Computing (at Stanford and Berkeley) by Dave Patterson and others. Can you design services to restart fast, maybe by keeping log information? If that really works well, then you can design your entire system around the principles of recovery and restart. If you don't like the behavior or performance of a service, you can kill it and let it restart. If it can be functioning again in a minute, this systems design approach can be very effective.

30:40: CAP Conjecture [slide 8]

30:40: At Amazon, all data applications are dominated by this theorem. Traditional data applications assumed that if you stored something in a database it would never go away [Durability, the "D" in the ACID properties]. In reality, because of redundancy, many nodes can be working in parallel and storing information, and then bad things can happen.

32:00: A Clash of Cultures [slide 5]

32:00: There's nothing wrong with transactions, they create a nice clean programming paradigm, and are good for programmers. But you must design for failure cases, because transactions can fail. So the ACID properties are great if you can get them, but getting these guarantees is costly. It may be fine if only a single node accesses the data. but not if 10's or 100's of machines need access.

[33:30] In that case another, more fuzzy, approach may be better -- BASE, in which data is basically available, more or less. Applications maintain a soft state, in which data is eventually consistent.

34:05: ACID vs BASE [unnumbered]

34:05: ACID has a pessimistic behavior, it will fail if it cannot reach the guarantees that you want. Availability is less important than consistency. For BASE systems, availability is the most important, and you are willing to sacrifice something to ensure it. So, for example, in a Web application you will design an application to accept and store shopping-cart input from a customer, and deal with minor problems in the data later. It's a weaker level of consistency, but you never want to tell the customer you can't accept their input.

35:50: Why the Divide? [Slide 7]

35:50: CAP stands for Consistency, Availability, and Partitioning. Eric Brewer came up with the conjecture that systems can only possess two of these three characteristics, which was subsequently proved to be true. That means systems designers must make choices, they must decide how to handle data reads and writes. If you insist on always enforcing consistency, your system may have to reject data interactions, making the system (in effect) unavailable at certain times. If you value availability and want to always accept user interactions, your application must then deal with the fact that some of its responses may later turn out to have been inconsistent.

38:45: Sometimes applications can deal with this. In the Web environment, the common technique of customer stickiness may help you to operate with lower levels of consistency. Once they have begun a session, customers are typically redirected to the same server cluster, or data center. So a local level of data consistency is sufficient; global consistency is unnecessary.

39:30: Consistency and Availability [slide 9]

39:30: Many applications have a workflow behavior. First the customer interacts with the shopping cart. At this time, availability is the most important. After that you do all kinds of things with that data, and during those activities, consistency is the most important. Now, because you are not interacting with the customer directly, if you can't obtain consistency for one data item, you can move on to process a different data item and come back later. Then you get to the shipment and delivery phase, and at this time the database is mostly read only. A data architecture that forces you to use the same powerful tool -- like a big relational database -- for all these different activities is not ideal. If you select data storage solutions that are appropriate for each phase, it's easier to scale your solution.

44:00: Partition-Tolerance and Availability [slide 10]

44:00: It's hard to program for weaker levels of consistency. Amazon has developed some API's, but Werner had no time to discuss these solutions. Slide 10 lists some examples, which he discussed briefly. The core design approach involves guaranteeing the durability of data inputs while relaxing consistency enforcement, then returning later to deal with any inconsistencies.

45:40: Techniques [slide 11]

45:40: Read the slide, because Werner does not talk about it!

We used to use a lot of DB technology at Amazon. It works really well, especially if most applications manipulate single data records using their primary keys. You can still create accessors that iterate over the entire database, but these should be relegated to a lower priority, background, status. The primary interfaces should offer only simple get/put accesses. In a production database that supports transactions, you don't need to also support queries, especially if the data is just XML text anyway. If engineers know what's inside those XML records, they may start coding against it!

48:00: Whatever DM software you are running, databases that require high performance need specialists to configure, operate, and manage them. Engineers can't do it effectively, you need DBAs, even for very simple access patterns. [Guideline 19.5 in High-Performance Client/Server: Although DBMSs may offer similar features, implementations usually differ. Never assume that a design rule of thumb learned for one DBMS can be applied unchanged to another.]

50:30: What does this mean for the data architecture?

50:30: Again, read the slide, because in the edited presentation stream, Werner appears to be speaking to a different slide altogether. And then he ran out of time, so ...

... that's it. A really insightful and informative talk by Werner Vogels. No doubt his presentation could have been improved, given more time, or even just better use of the time available. But all the same, it is very stimulating (I think) and well worth several listens, until you grasp the central points -- all of which I agree with. In fact, Werner's conclusions circle back to the conclusion of my book, and my opening statement in the first post in this series:

Documenting this talk has been both educational and satisfying. But -- since I can't type nearly as quickly as Werner can talk -- I may have misquoted him somewhere. If you spot a mistake please let me know, and I'll correct it.

This series of posts contains some material first published in High-Performance Client/Server. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

Tags: Werner Vogels, Amazon, QCon, distributed systems, asynchronous architecture, Web services, SOA, performance, scalability, synchronization, autonomy, multi-transaction, workflow, David Patterson, Recovery Oriented Computing, Fred Brooks, Mythical Man Month, Eric Brewer, ACID properties, two-phase commit, BASE, CAP theorem, performance matters

Always assume high latency, not low latency

This is the third in a series of posts presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment.

The first reviewed the case for asynchronous communication among interdependent components or services, and Bell's Law of Waiting. The second highlighted The Fallacies of Distributed Computing, and discussed the importance of reflecting the business process in distributed systems design.

This post reviews The Challenges of Latency, an article about how asynchronous architectures can improve the quality of Web applications, published on the InfoQueue site by eBay architect Dan Pritchett in May 2007. Dan's article is especially relevant today, given the high level of interest in adopting Web services and SOA approaches.

Dan explains why global, large-scale architectures need to address latency, and what architectural patterns can be applied to deal with it. He begins by invoking the second fallacy of distributed computing:

Latency.

The time it takes packets to flow from one part of the world to another. Everyone knows it exists. The second fallacy of distributed computing is "Latency is zero". Yet so many designs attempt to work around latency instead of embracing it. This is unfortunate and in fact doesn't work for large-scale systems. Why?

In any large-scale system, there are a few inescapable facts:

1. A broad customer base will demand reasonably consistent performance across the globe.
2. Business continuity will demand geographic diversity in your deployments.
3. The speed of light isn't going to change.

The last point can't be emphasized enough. The speed of light dictates that even if we can route packets at the speed of light, seems unlikely, it will take 30ms for a packet to traverse the Atlantic.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Latency hurts customer service

He emphasises the connection between Internet latency and customer service:

The Internet is a part of foundation of the global economy.

Companies need to reliably reach their customers regardless of where they may be located. Architectures that force close geographic proximity of the components limit the quality of service provided to geographically distributed customers. Response time will obviously degrade the further customers are from the servers, but so will reliability. Despite the tremendous increase in the reliability of traffic routing on the Internet, the further you are from a service, the more often that service will be effectively unavailable to you.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Latency tolerance

After spelling out the principle that I have highlighed above as today's Performance Wisdom, he goes on to make the case for introducing asynchronous interactions as the way to achieve latency tolerance.

The web has created an interaction style that is very problematic for building asynchronous systems. The web has trained the world to expect request/response interactions, with very low latency between the request and response. These expectations have driven architectures that are request/response oriented that lead to synchronous interactions from the browser to the data. This pattern does not lend itself to high latency connections.

Latency tolerance can only be achieved by introducing asynchronous interactions to your architecture.

The challenge becomes determining the components that can be decoupled and integrated via asynchronous interactions. An asynchronous architecture is far more than simply changing the request/response from a call to a series of messages though. The client is still expecting a response in a deterministic time. Asynchronous architectures shift from deterministic response time to probabilistic response time. Removing the determinism is uncomfortable for users and probably for your business units, but is critical to achieving true asynchronous interactions.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Web Services in SOA - Synchronous or Asynchronous?, a paper by Torry Harris Business Solutions, offers another introduction to the pro's and con's of synchronous and asynchronous architectures.

Dan accepts that not all Web application components can be designed to function asynchronously, but argues that designers can identify those use cases that do support synchronous interactions. These arguments confirm my earlier conclusion that synchronous solutions must be combined with asynchronous designs in which the user must accept that unconfirmed changes will be reflected in the enterprise database(s) at a later time.

Data partitioning

He makes a very good point about the importance of designing data distribution from the outset:

You can decompose your applications into a collection of loosely coupled components; expose your services using asynchronous interfaces, and yet still leave yourself parked in one data center with little hope of escape. You have to tackle your persistence model early in your architecture and require that data can be split along both functional and scale vectors or you will not be able to distribute your architecture across geographies.

I recently read an article where the recommendation was to delay horizontal data spreading until you reach vertical scaling limits. I can think of few pieces of worse advice for an architect. Splitting data is more complex than splitting applications.

But if you don't do it at the beginning, applications will ultimately take short cuts that rely on a monolithic schema. These dependencies will be extremely difficult to break in the future.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

ACID or BASE?

Noting that the traditional way to maintain database consistency across partitioned data requires ACID-compliant distributed transactions and two-phase commit protocols, Dan advocates a (cleverly-named) alternative to the ACID properties, the BASE approach to database consistency:

The problem with distributed transactions is they create synchronous couplings across the databases. Synchronous couplings are the antithesis of latency tolerant designs. The alternative to ACID is BASE:

Basically Available
Soft state
Eventually consistent

BASE frees the model from the need for synchronous couplings. Once you accept that state will not always be perfect and consistency occurs asynchronous to the initiating operation, you have a model that can tolerate latency.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007

Another article worth reading, Web-Services Transactions by Doug Kaye, advances similar arguments without using the BASE terminology.

References: business-driven or event-driven architectures

While these articles present, at a high-level, a convincing case for asynchronous architectures, many others have elaborated on the implementation details. Here are five examples of more detailed treatments, in approximately descending order of generality:

• The wikipedia article on Event-driven architecture.
• Event-Driven Architecture Overview, a very detailed post by Brenda Michelson in her blog, Elemental Links. [A blog response from Doug McClure exemplifies the gulf between the concerns of business/application architects and those of IT/network management, despite all the talk of "alignment" these days. As I wrote in my book, "Middleware is the reason why network specialists and application programmers cannot communicate!" (Guideline 15.3, p469).]
• Event-driven services in SOAby Jeff Hanson, JavaWorld.com, January 31, 2005.
• Message Queuing Applications, Microsoft Developer Network (MSDN).
• Developing Asynchronous Web Services with Java Message Service by Rico Cruz and Marina Sum, Sun Developer Network.

Next: the CAP theorem

Dan points out that adopting the BASE approach to consistency forces you to understand a very important principle, known as The CAP Theorem. This theorem was first propounded in a 1998 presentation -- Lessons from Internet Services: ACID vs. BASE -- by Dr. Eric Brewer of Inktomi, now a professor at UC Berkeley.

Of course there are situations where data needs to be consistent at the end of an operation. The CAP Theorem is a useful tool for determining what data to partition and what data must conform to ACID.

The CAP Theorem states that when designing databases you consider three properties, Consistency, Availability, and Partitioning. You can have at most two of the three for any data model.

Organizing your data model around CAP allows you to make the appropriate decisions with regards to consistency and latency.

-- Dan Pritchett, The Challenges of Latency, May 2, 2007 [emphasis added]

Because The CAP Theorem plays a crucial role in the design of large scalable systems using asynchronous architectures, I will be devoting my next post to it.

This series of posts contains some material first published in High-Performance Client/Server. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

Tags: distributed systems, asynchronous architecture, Web services, SOA, middleware, serialization, synchronization, queues, decoupled processes, multi-transaction, workflow, distributed computing, Dan Pritchett, eBay, Brenda Michelson, Elemental Links, Doug McClure, Eric Brewer, Inktomi, ACID properties, two-phase commit, BASE, CAP theorem, application performance, scalability, performance wisdom, performance matters

This post is the second in a series presenting arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment. The first post reviewed the general case for asynchronous communication among interdependent components or services, and highlighted Bell's Law of Waiting.

In this post I discuss how the design of distributed systems should draw on that of manual business systems. Of course, distributed computing can shorten the timescales of some business operations enormously. But drawing analogies with the way manual systems work is an observation that will help us to design efficient and scalable distributed systems.

In Five Scalability Principles, I reviewed an article published by MySQL about the five performance principles that apply to all application scaling efforts. When discussing the first principle -- Don't think synchronously -- I stated that:

Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance (distributed) systems.

Starting from the wrong place

Designing a computer system based on multi-transaction workflows is not a particularly revolutionary proposal. Indeed, if we had somehow been able to skip the first 40 years of the computer age and start automating our business systems using today’s technology, we would probably not have thought it the least bit unusual, because all manual systems work this way.

But like our computers, our reasoning can be so logical that it lacks real thought. Occasionally, we need to balance our linear thinking with a small dose of simple wit like that ascribed to the Irish farmer, who, when asked by strangers for directions to a distant town, began his answer by saying “Well, I wouldn’t start from here”!

Most of our troubles stem from the fact that, when trying to reach the destination of distributed systems, we keep starting from the wrong place, namely the application designs and systems software of centralized computing:

• Design discussions dwell on how best to partition applications for the distributed environment. Manual systems, however, are already partitioned naturally -- the supposedly monolithic application that is being “partitioned” would not exist in the first place if it had not been conceived as “the right solution” by a designer with a centralized computing mindset.
• The reason computer science devoted so much attention to distributed databases and distributed transaction management is because these concepts are extensions of the core mechanisms of centralized information processing -- shared databases and transaction monitors.

But The Fallacies of Distributed Computing -- assembled during the 1990's by architects at Sun and the subject of today's Performance Wisdom (above) -- highlight crucial differences between centralized and distributed computing. Adding network components to an application introduces many potential problems that a centralized solution does not have to consider. So rather than trying to force the centralized mechanisms to work in a distributed environment, we should adopt mechanisms that are more appropriate.

Starting from the right place

In fact, we should start from the design of manual business systems. All large scale human systems are inherently distributed and asynchronous in nature. Even the participants in close knit team efforts operate asynchronously. We find chorus lines, cheerleaders, marching bands, and synchronized swimmers so interesting because they are such an aberration. So, before the advent of the centralized mainframe, the idea of recording an entire business transaction with a single synchronized set of human actions did not arise because it is so absurdly impossible.

Traditionally, the business process is divided into its natural components (or phases), according to the roles of the various human processors (or workers). Work flows through the phases, and information is recorded as necessary along the way. If anything goes wrong along the way, the appropriate set of compensating actions must be taken to undo whatever partial progress has been made. And the whole operation is designed to ensure that no irrevocable actions are taken too early in the process--usually meaning before the money is in the bank. Companies that mail out the diamonds before cashing the checks soon learn how to design a more effective multi-phase business process.

Asynchronous architectures may reflect manual systems

Manual systems always permit asynchronous operation of their separate components, because no other mode of operation is possible. Only computers make synchronous changes even possible. To a degree, centralized computing could deliver synchronous changes to related databases because the same computer managed all the system resources. Peaks in the workload could cause contention and delays, but provided the machine kept running, a congested machine acted as it’s own governor.

But processor technology does not allow the centralized computing model to scale up without limits. When the workload surpasses the capabilities of the largest centralized processor, the only way to keep growing is to divide and conquer -- to create a network of computers. Networked computers demand a different approach to application design. Pursuing the vision of enterprise wide synchronization of information through networked computers can cause delays and inefficiencies when any part of the whole system operates below par. This is the Achilles heel of interdependent systems--the whole is no stronger than it’s weakest part.

Therefore, the rigid concept of synchronous transactions must be replaced by a wider range of possible application designs:

• The older, synchronous methods are still appropriate for changes that are within the scope of a single processor, or even for occasional communication between controlled components separated by a very carefully controlled, high speed network;
• These must be combined with asynchronous designs in which the user must accept that unconfirmed changes will be reflected in the enterprise database(s) at a later time.

Business process design

When we do a good job of distributed systems design, it becomes an integral part of business process design. Rather than bending the business process to meet the needs of a centralized computer, we must blend the power of distributed computers into the business process.

Ironically, these changes bring us almost full circle back to the days of manual systems. In a manual system, it is normal for changes to be recorded quickly in one location, but for those changes to take a few days to percolate through the system, with the total processing time being somewhat uncertain. Distributed computing shortens the timescales, but applying many design principles that made manual systems work efficiently can help us to create effective and scalable distributed systems, even when we do not control the performance characteristics of all the components of those systems.

In my next post I will review some recent thinking about the intersection of asynchronous architectures and the world of Web services and SOA.

This post contains material first published in High-Performance Client/Server, Chapter 16: Architecture for High Performance, pp509-510 and 526-527. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

Tags: distributed systems, asynchronous architecture, serialization, queues, decoupled processes, multi-transaction, workflow, performance wisdom, distributed computing, fallacies, Peter Deutsch, James Gosling, Bill Joy, Tom Lyon, Sun, software performance, scalability, performance matters

All computers wait at the same speed

In Five Scalability Principles, I reviewed an article published by MySQL about the five performance principles that apply to all application scaling efforts. When discussing the first principle -- Don't think synchronously -- I stated that Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance (distributed) systems.

That's a quote from High-Performance Client/Server, from a section on Abandoning the Single Synchronous Transaction Paradigm, in Chapter 15, Architecture for High Performance. My 1998 book is out of print now, and contains some outdated examples and references. But most of the discussions of performance principles are timeless, and you can pick up a used copy for about $3.00 at Amazon.

So I am planning some more posts built around excerpts from the manuscript. I'll be updating and generalizing the terminology as necessary for today's environments, and adding some guidelines in my Performance Wisdom series.

Asynchronous architectures are more scalable

The first posts will elaborate on the arguments for asynchronous architectures as the optimal way to build high-performance, scalable systems for a distributed environment. I begin by reviewing the general case for asynchronous communication among interdependent components or services.

In the typical distributed enterprise, there will inevitably be fluctuations in the distribution of work to be done, as business volumes rise and fall, and fluctuations in the availability of network and processing resources.

Even if we have designed our systems to accommodate peak processing volumes, it is normal for some servers, some application components, or some part of a large network to be out of action some of the time. Therefore, if we design applications that require all resources to be available before we can complete any useful work, we reduce the availability of the whole system to the level of its most error prone component.

For optimal performance, we should design applications to accommodate unexpected peaks in the workload, server outages, and resource unavailability. This means application and system design must:

• Emphasize concurrent operation in preference to workload serialization
• Prefer asynchronous connections to synchronous ones between clients and servers
• Place requests for service in queues and continue processing, rather than waiting for a response
• Create opportunities for parallel processing of workload components
• Distribute work to overflow servers to accommodate peak volumes
• Provide redundant servers to take over critical workload components during peaks and outages

Design applications that don’t wait

Each of these topics is too large to review in any detail in this post, but their central theme can be summed up as: Design applications that don’t wait. Note an important distinction between the behavior of individual transactions or units of work, and the behavior of the system as a whole. Individual transactions may indeed have to wait until they can obtain the processing resources they need. But the application as a whole should continue processing, with a minimal allocation of resources to any transactions flowing through the system.

That way, the scarce computing resources of one server do not sit idle waiting for delayed transactions to receive responses from other services or components. For example, here's some advice from BEA, taken from Best Practices for Application Design when programming the WebLogic Java Message Service (JMS):

Asynchronous vs. Synchronous Consumers

In general, asynchronous (onMessage) consumers perform and scale better than synchronous consumers:

• Asynchronous consumers create less network traffic. Messages are pushed unidirectionally, and are pipelined to the message listener. Pipelining supports the aggregation of multiple messages into a single network call.
• Asynchronous consumers use fewer threads. An asynchronous consumer does not use a thread while it is inactive. A synchronous consumer consumes a thread for the duration of its receive call. As a result, a thread can remain idle for long periods, especially if the call specifies a blocking timeout.
• For application code that runs on a server, it is almost always best to use asynchronous consumers (which) prevents the application code from doing a blocking operation on the server. A blocking operation, in turn, idles a server-side thread; it can even cause deadlocks. Deadlocks occur when blocking operations consume all threads. When no threads remain to handle the operations required to unblock the blocking operation itself, that operation never stops blocking.

Bell's Law

In conclusion, when designing distributed systems, we should always recall Tom Bell's humorous observation, All computers wait at the same speed. A computing resource that is waiting to be used, especially a processor, is a wasted resource.

This post contains material first published in High-Performance Client/Server, Chapter 11: The Sharing Principle, p360. Tags: distributed systems, asynchronous architecture, serialization, queues, Bell's Law, Tom Bell, performance wisdom, software performance, scalability, performance matters

Don’t think synchronously, ...

The 12 Days of Scale-Out is a section of the MySQL site. It consists of a series of twelve articles, eleven of which are case studies describing large-scale MySQL implementations. But Day Six is a bit different -- it spells out five fundamental performance principles that apply to all application scaling efforts.

This subject is vitally important to MySQL, whose server replication and high availability features ... allow high-traffic sites to horizontally 'Scale-Out' their applications, using multiple commodity machines to form one logical database -- as opposed to 'Scaling Up', starting over with more expensive and complex hardware and database technology.

I know from first-hand experience that these claims are valid. At Keynote, my team used MySQL as the foundation for the Performance Scoreboard. In this data mart application, MySQL supports supports the continuous insertion of new measurements at the rate of several million per day, plus hourly aggregation into summary tables, plus the queries needed to support continually updated dashboard displays for every customer, plus any ad hoc queries generated by customers doing diagnostic investigations.

Learning the hard way

According to the article, MySQL's database experts ... have seen many companies fall into a few common traps when they first design their systems, only to run into performance issues once the explosive growth hits. So, adopting the anti-pattern approach to providing guidance, the article presents the principles as the Top Five Scale-Out Pitfalls to Avoid. These are:

• Don’t think synchronously
• Don’t think vertically
• Don’t mix transactions with business intelligence
• Avoid mixing hot and cold data
• Don’t forget the power of memory

It is common for people to get smarter about performance when they have to find and fix problems. I learned about database performance first-hand between 1970 and 1995, while designing and tuning IBM database systems -- first IMS, and then DB2. In the process I discovered -- often the hard way -- that all large computer systems are subject to the same principles. And over the years I ran across other authors and teachers who (not surprisingly) had discovered the same things. Some had even invented memorable and insightful ways of describing their insights as "laws" or "rules".

When I wrote High-Performance Client/Server, I tried to capture these clever sayings as numbered guidelines. So in this post, I'm going to reprint each of MySQL's five scale-out pitfalls, followed by the corresponding guidelines and some related excerpts from the manuscript of that book (marked as "HPCL").

1. Don’t think synchronously

Thinking synchronous is the single biggest mistake in architecting a Scale-Out design. Generally, when load is added to an already-loaded system, some part of the system will become a bottleneck -- and response times will increase. In scale-out, with a large system consisting of multiple machines, thinking synchronously will add a lot of wait time and hurt performance. Any truly large scale-out design will have to introduce asynchronous communication, parallelization, and strategies to deal with approximate or slightly outdated data.

--Top Five Scale-Out Pitfalls to Avoid, The 12 Days of Scale-Out, Day 6

Abandoning the Single Synchronous Transaction Paradigm

HPCL: The ... concept of a heterogeneous distributed database with synchronized updates is a vision of utopia that swims against the tide of computing technology. The tight controls over application processing that are possible on a mainframe are incompatible with many aspects of the move to widespread distributed processing... Decoupled processes and multi-transaction workflows are the optimal starting point for the design of high-performance enterprise client/server systems:

• Decoupled processes. Decoupling occurs when we can separate the different parts of a distributed system so that no one process ever needs to stop processing to wait for the other(s). The driving force