Comments on: More on Redesigns: Only 30% of Web Site Changes Have a Positive Impact

By: scottporad

scottporad — Fri, 02 Apr 2010 05:41:39 +0000

@doug That’s an interesting point, but I’m not sure why testing conservatively is a bad thing. It seems to me that 5% of wasted effort is better than 70%.

I suppose you could say it’s a risk-reward equation–if we take bigger risks, they might fail more often, but will have bigger wins. I don’t know about that…how could we validate that theory?

By: doug

doug — Tue, 30 Mar 2010 08:59:18 +0000

Nice post.

I wonder whether 30% might be an optimal figure, or close to it. For instance, if our figure was 30% and three months and a hundred tests later it increased to 95%, i don’t think i would consider that an improvement–instead probably that we are testing too conservatively.
–doug

By: Sometimes the 25th time is the charm at ScottRu

Sometimes the 25th time is the charm at ScottRu — Mon, 15 Mar 2010 16:05:08 +0000

[...] a single test produced a meaningful improvement. (I noted this a few months ago on Scott Porad’s blog post about redesign testing.) Our best-guess UI has either outperformed or shown no statistical difference. That’s kind [...]

By: Scott Ruthfield

Scott Ruthfield — Fri, 13 Nov 2009 06:13:00 +0000

I’m not willing to concede defeat just yet. Maybe I’m just at a local maxima and I’m not experimenting wildly enough; maybe I just haven’t hit the right combination yet.

Clearly there is a tipping point where everything you try can’t be meaningfully better than your best option, but (especially when you make trying easy) it never hurts to experiment.

By: scottporad

scottporad — Fri, 13 Nov 2009 03:56:20 +0000

Scott…that is really, really fascinating.

To me, it seems like there is something to be learned from the fact that no matter what you do conversion doesn’t go up. Perhaps there is only so much a site can do to drive conversions…the remainder has to do with the product and product/market fit.

Thoughts?

By: Scott Ruthfield

Scott Ruthfield — Thu, 12 Nov 2009 20:09:38 +0000

Scott – your post made me want to share something interesting.

I have a site that does ~50K uniques/month, built with a partner, where we did a lot of design ahead of time and launched it. Now about a year later, we have some time and we’ve run ~20 A/B and multivariate tests. We keep having great ideas based on what we think we’ve learned.

Guess what? In every single test we’ve run, the control has beaten (or tied) the variations. We have not yet succeeded in increasing conversions with any change we’ve made. Seems like our first instincts, based on no real data on usage, were at least as good as our later instincts.

From this perspective, optimization testing works, though it doesn’t make for as good a story.