More on Redesigns: Only 30% of Web Site Changes Have a Positive Impact
Someone challenged me about yesterday’s post where I said:
All of these little changes cost time to develop, rarely are beneficial, and often harmful.
Let me give you a little data to back that up: a lot of people think they know what will make their site better, but they really don’t.
See, you need to remember this: if you work on a web site you look at it very, very differently than your users. First, you look at the thing all day long and are familiar with every nook, cranny and blemish. Second, you’re “in the biz” which means your experience with the web, and with the site is not even remotely similar to the ordinary user. Combined, the chances that you can redesign your web site and make it better for your users isn’t inconceivable, but they’re not in your favor.
Assume that by simply guessing what your users want—that by flipping a coin—you have a 50/50 chance of getting a big redesign right. Those are pretty lousy odds. What you think you can do better than that? If so, think again…
Ron Kohavi is The Man when it comes to online analytics. Currently, he runs analytics for Microsoft, and previously did so at Amazon. If you know anything about building top-tier web sites then you know that Amazon was the early leader in the web testing space.
Ron presented at the Seattle Tech Startups meeting in September (slides, video) where showed data that analyzed thousands of A/B tests. The results confirmed my overall experience from an entire career of web development: only about 30% of changes to a web site have a positive impact, roughly another third are neutral, and the remainder are harmful.
If you didn’t grok that last sentence, let me summarize it for you again:
60-70% of the changes that happen on a web site are either useless, or worse, harmful.
So, my point here is this: as web site developers, we’re better off making small, incremental changes that we can measure and verify because the likelihood of our success is low. Although one commenter on a previous post indicated that his redesign vastly improved site performance, according to a broad collection of data, this typically isn’t the case.
Scott – your post made me want to share something interesting.
I have a site that does ~50K uniques/month, built with a partner, where we did a lot of design ahead of time and launched it. Now about a year later, we have some time and we’ve run ~20 A/B and multivariate tests. We keep having great ideas based on what we think we’ve learned.
Guess what? In every single test we’ve run, the control has beaten (or tied) the variations. We have not yet succeeded in increasing conversions with any change we’ve made. Seems like our first instincts, based on no real data on usage, were at least as good as our later instincts.
From this perspective, optimization testing works, though it doesn’t make for as good a story.
Scott Ruthfield
12 Nov 09 at 12:09 pm
Scott…that is really, really fascinating.
To me, it seems like there is something to be learned from the fact that no matter what you do conversion doesn’t go up. Perhaps there is only so much a site can do to drive conversions…the remainder has to do with the product and product/market fit.
Thoughts?
scottporad
12 Nov 09 at 7:56 pm
I’m not willing to concede defeat just yet. Maybe I’m just at a local maxima and I’m not experimenting wildly enough; maybe I just haven’t hit the right combination yet.
Clearly there is a tipping point where everything you try can’t be meaningfully better than your best option, but (especially when you make trying easy) it never hurts to experiment.
Scott Ruthfield
12 Nov 09 at 10:13 pm
[...] a single test produced a meaningful improvement. (I noted this a few months ago on Scott Porad’s blog post about redesign testing.) Our best-guess UI has either outperformed or shown no statistical difference. That’s kind [...]
Sometimes the 25th time is the charm at ScottRu
15 Mar 10 at 9:05 am
Nice post.
I wonder whether 30% might be an optimal figure, or close to it. For instance, if our figure was 30% and three months and a hundred tests later it increased to 95%, i don’t think i would consider that an improvement–instead probably that we are testing too conservatively.
–doug
doug
30 Mar 10 at 1:59 am
@doug That’s an interesting point, but I’m not sure why testing conservatively is a bad thing. It seems to me that 5% of wasted effort is better than 70%.
I suppose you could say it’s a risk-reward equation–if we take bigger risks, they might fail more often, but will have bigger wins. I don’t know about that…how could we validate that theory?
scottporad
1 Apr 10 at 10:41 pm