Someone challenged me about yesterday’s post where I said:
All of these little changes cost time to develop, rarely are beneficial, and often harmful.
Let me give you a little data to back that up: a lot of people think they know what will make their site better, but they really don’t.
See, you need to remember this: if you work on a web site you look at it very, very differently than your users. First, you look at the thing all day long and are familiar with every nook, cranny and blemish. Second, you’re “in the biz” which means your experience with the web, and with the site is not even remotely similar to the ordinary user. Combined, the chances that you can redesign your web site and make it better for your users isn’t inconceivable, but they’re not in your favor.
Assume that by simply guessing what your users want—that by flipping a coin—you have a 50/50 chance of getting a big redesign right. Those are pretty lousy odds. What you think you can do better than that? If so, think again…
Ron Kohavi is The Man when it comes to online analytics. Currently, he runs analytics for Microsoft, and previously did so at Amazon. If you know anything about building top-tier web sites then you know that Amazon was the early leader in the web testing space.
Ron presented at the Seattle Tech Startups meeting in September (slides, video) where showed data that analyzed thousands of A/B tests. The results confirmed my overall experience from an entire career of web development: only about 30% of changes to a web site have a positive impact, roughly another third are neutral, and the remainder are harmful.
If you didn’t grok that last sentence, let me summarize it for you again:
60-70% of the changes that happen on a web site are either useless, or worse, harmful.
So, my point here is this: as web site developers, we’re better off making small, incremental changes that we can measure and verify because the likelihood of our success is low. Although one commenter on a previous post indicated that his redesign vastly improved site performance, according to a broad collection of data, this typically isn’t the case.