When an analysis of performance data from Keynote Systems and Symphoniq Corporation was conducted, the results showed that performance problems could be costing almost $1 billion in lost e-commerce revenues.
Furthermore, a comparison of the statistics from these two companies reveals that the current standard of performance measurement, ´synthetic´ transaction monitoring, may be understating the impact of performance issues. Synthetic transaction monitoring, the process of using scripted transactions to periodically simulate end-users performing business transactions such as logging in, searching, and purchasing, creates a fa
lse sense of security for leading online merchants.
The Magnitude of Soft Downtime ´ Poor Performance and Sales
Downtime gets a lot of attention. As one of my clients commented, ´My job is to keep our company´s name out of the papers. When we go down, we make the front page of the Wall Street Journal.´
Yet despite the publicity that ´hard´ downtime receives, its impact may be dwarfed by the ´soft´ downtime of poor performance. Who hasn´t abandoned a Web site because of slow page load times? Yet according to standard measures of availability, those incidents don´t count.
In 2001, Zona Research reported that 30% of surfers would abandon a Web site if it took more than 8 seconds to load, creating the widely-quoted ´8-second´ standard.
Four years have cut that safety margin in half. ´The wait becomes frustrating for most Internet users after about 4 seconds,´ said Mark Ehr, an analyst with Enterprise Management Associates, ´Patience is waning as more shoppers become accustomed to quicker broadband connections.´
Let´s assume that 30% of surfers abandon a site that takes more than 4 seconds to load . This gives the following formula for the impact of site performance problems on revenues:
To calculate the impact of performance problems, I turned to two companies that provide Web performance reports, Keynote Systems and Symphoniq Corporation.
Keynote Systems publishes weekly performance indices for various sectors, ranging from travel and hospitality to financial services. I looked at the following sectors, totaling 63 Web sites: Travel & Hospitality, Broker, E-commerce, and News Media, along with site performance data collected by Symphoniq Corporation during the 2004 holiday shopping season. Symphoniq audited the performance of 60 e-commerce Web sites from Thanksgiving through December 26, for sites ranging from Alaska Airlines to Walmart.com.
Then, I cross-referenced the two lists to generate a list of 25 common sites.
I began by looking at Keynote´s measure of availability, which is considered the industry standard. Keynote uses a global network of reference sites which monitor Web sites using simulated user transactions. The performance indices studied measure availability from the 10 largest metropolitan areas in the United States. The avera
ge success rate for completing transactions across all those sectors was 98.59%, which means that the percent downtime was 1.41%.
If you input this into the Impact equation, you find that the impact is ($15.8 billion) * (1.41%), or $223 million dollars.
To measure the impact of sluggish Web site load times, I next turned to Symphoniq´s data. Symphoniq monitored a broad set of pages per site (at least 100 URLs) and reported on the percentage of pages that took over 4 seconds to load.
Across the 25 sites in common, the average proportion of Web pages that took longer than 4 seconds to load was 13.8%. The individual Web site totals ranged from 0.56% to 52.09%.
Inputting this 13.8% into the Impact equation, I find that the impact of slow page load times is ($15.8 billion) * 0.3(13.8%), or $654 million dollars.
By combining these two measures, I can estimate that the total impact of Web performance problems on the 2004 holiday shopping season was $877 million.
A False Sense of Security?
Even though downtime and outages receive the bulk of the press, the model showed that the soft downtime of slow Web sites represented the majority of the real business impact. Slow pages cost e-tailers three times as much as unplanned outages, representing nearly 74% of the billion-dollar shortfall.
What this indicates is that the current standard of performance measurement, single-transaction synthetic monitoring, while useful for benchmarking and trending, fails to capture the true impact of Web performance problems. In fact, the synthetic transaction approach may be giving e-tailers a false sense of security.
To explore this issue further, I examined the performance results for each of the 25 common sites.
In 92% of cases, Keynote's single-transaction availability approach resulted in a lower estimate of problems. In 76% of cases, Keynote underestimated user-visible problems by more than 80%. Overall, slow pages occurred nearly 10 times as often as outages, 13.8% versus 1.41%.
To some extent, this is comparing apples and oranges; Keynote measures availability, while Symphoniq measures Web site speed. Nonetheless, the analysis underscores the point that looking at availability figures for a small number of URLs fails to capture the reality of the customer experience. If customers become frustrated with a Web site´s performance and leave without completing a transaction, the effect is the same as if the site went dark´the e-tailer loses revenues and customers.
Conclusions: The Billion-dollar Cost of Poor Site Performance
Web site performance is a billion-dollar problem. Downtime and slow-loading pages cost e-tailers $877 million between Thanksgiving and December 26, 2004, and probably cost billions more during the rest of the year. The majority of the revenue impact comes not from Web site outages, but rather from the prevalence of slow Web pages and the impatience of Web customers in a broadband world.
While the standard ´hard´ measures of availability are still important, their importance is being eclipsed by the importance of measur ing page load times and other ´soft´ performance issues. I recommend that Web operations adopt tools that allow them to measure the experience of real users across all URLs to understand the real business impact of poor performance.
Tom is co-founder of, TargetFirst Network, a value-based online advertising network and author of dozens of articles on numerous online business topics. He has also written or co-written three books, including the best selling Web Advertising and Marketing.