Every year, more and more shoppers turn to the Web for their holiday shopping, with total sales in 2006 projected to be in the multi-billion dollar range. But will online retailers be up to the task? Our recent study suggests that many will not.
by Ben Rushlo
My team at Keynote recently studied 25 top online retailers in three categories: Books and Music, Electronics and Apparel. The study involves measuring a typical customer's navigation path through each site -- from Home Page, to Search, to Product Details, and finally Checkout.
Using computers ("agents") that emulate the behavior of a customer using an IE browser, we measured the exact same sequence of steps on each site from 10 locations throughout the US, over various connection speeds, every hour for a month (mid-May through mid-June, 2006). This produces a large data sample (over 6,000 data points per site), which we then subject to a lot of statistical analysis. By the time we publish the report, we believe we have a reliable picture of a site's health and readiness.
This is the second year we've conducted this particular study, and the 2006 results were surprising. While the top-ranked sites continue to provide excellent service -- almost perfect availability, excellent download speeds, and very little inconsistency -- the lowest-ranked sites had some serious failings. Without doubt, the performance problems we saw were bad enough to dissatisfy customers and impact the bottom line.
The problems fall into three general areas:
We consider a transaction (the sequence of steps) to be unavailable if any part of the purchase path fails so that the customer is unable to complete their transaction. And when 30% or more of our measurements during an hour report a site as unavailable, we count it as an outage.
Only 4 of the 25 measured sites registered no outages during the month, while some of the least reliable sites had more than 15 hours of downtime. These hours are during the peak period each day (8 a.m. to midnight EDT). Apparel sites were especially outage-prone, averaging over 4 outages each.
We all know how frustrating it can be to try to buy something online only to have the process fail midway through, or to not even be able to get to the site's Home Page. I'm not sure which is the more annoying. And if these major retail sites can't stay up under relatively light summertime loads, how will they respond when their traffic increases dramatically during the holiday season? In the increasingly competitive retail marketplace, being down for even a single hour at that time of year can have a significant financial and brand impact.
2. Load Handling
One area of our analysis involves a site's ability to keep up with its current load, without any performance degradation. On a site that is built with sufficient capacity, performance does not change noticeably as traffic fluctuates during the day. At this time of year, most retail sites should be idling, because traffic is light. Indeed, we saw that the best sites, Barnes and Noble and Gap, had virtually no slow-downs each day, indicating that they can handle their present load comfortably.
Of course, it takes a controlled load-testing project to discover what happens when traffic volumes are doubled, tripled, quadrupled, and so on. But judging from our measurements, these sites do appear well prepared to handle increases in their load. In contrast, several sites already display significant load-handling issues now, slowing down as much as 100% each day. For example, a page that normally appears in 3 seconds would take 6 seconds under load. Not only does this kind of sluggish behavior annoy current customers, it suggests that these sites are highly likely to crumble under the increased load, once holiday shoppers begin to fill their online storefront -- unless something is done before then to increase capacity.
3. Dial-Up Performance
While many of us can't remember the last time we used dial-up, a large percentage of users still use slower connections. These might not be dial-up, but a poor wireless connection, a slow hotel connection, or a satellite connection may not be much faster. So site designers shouldn't forget about their bandwidth-challenged customers.
Yet that's exactly what seems to be happening. Many retail sites seem set on alienating the dial-up user. The sites we tested averaged 30-40 seconds per page, which really adds up fast in a 10-page shopping transaction. And in some cases we even clocked Home Pages taking over 100 seconds to download. A few sites got it right, Dell being a good example. They have the fastest Home Page in the study for dial-up users because they serve a slightly trimmer page for this audience.
In short, the best sites are ready, and their performance sets the standard for the retail industry. But even among the leading retailers we studied, many could do a lot to improve the quality of the online shopping experiences they offer.
More importantly, if one of these sites is struggling now, what will happen in December, when I desperately need to buy my last-minute gifts? I'm not optimistic they will be ready to take my business. And I absolutely hate fighting the crowds at the local shopping mall, so -- along with thousands of others like me -- I'll be buying from their competitor's site.
Update #1: To hear a longer (12.5 min) discussion of these results and related issues in the retail industry, listen to a StorefrontBacktalk Week In Review Audiocast discussion I took part in. Click on the link for the section on whether the major E-Commerce sites are ready for the holiday rush or the holiday crash.
Update #2: I was also interviewed about the study today (8/25) on CNBC's Closing Bell program. As you might expect, that interview was a lot shorter than the StorefrontBacktalk Audiocast. You can view a replay online; the username is keynote and password is keynote0895.
[This post was first published on blogger on August 24, 2006.]