Below you’ll find a complete case study where we ran an A/B test within a shopping cart. We were testing the One Step Checkout Magento Extension against the standard Magento “One Page Checkout”. The results may surprise you 
In the words of Bryan Eisenberg, “Always be Testing” is a big motto around here. No matter what we’re doing, we’re always looking to improve our sites conversion rates. With that being said, I wanted to take a quick second to share some results about an A/B test we ran on an eCommerce site using the OneStepCheckout Magento Extension. (Learn more about the extension here.)
One of the perks of using an open source platform like Magento, is it opens all types doors for developers to make the product that much better. Seeing that the eCommerce website that we were testing was using Magento, it allowed us to do some very creative and robust conversion testing.
Getting the A/B Test Setup
For starters, setting up the test was a lot easier than we originally planned. Simply install the Magento Extension, setup the configuration as you’d like and you’re ready to roll. From there, you need to setup the A/B test to track the results. Using Google Website Optimizer, we just setup an A/B test to get the test up quick and start gathering data right away.
TIP: When testing areas throughout your site try to have a method to the madness. In other words, don’t just test something because it looks cool or it’s a lucky guess. At NetMediaGroup, we utilize all of our analytics to make educated decisions on what to test.
The 2 Variations
STANDARD MAGENTO ONEPAGE CHECKOUT:
ONE STEP CHECKOUT MAGENTO EXTENSION:
The Results (Shocking)
To my surprise it was detrimental. It was a negative outcome (-26%). Ouch. Now, let me make it clear though. After seeing these results it DOES NOT MEAN YOU SHOULDN’T TRY this Magento extension. There are several variables that go into account. The demographics that use our site, the type of product, the price point, etc. This was a fantastic solution for a shopping cart. We’re currently rolling out to other sites to see if we can replicate the results or prove our point that it was just the particular site we tested it on.
In conclusion, this is why it’s always best to test everything and yet another reason why we’re consistently testing. There’s always room for improvement.
If you’ve used this extension, I’d love to hear what your results were…
Hi, and thank you for the nice post!
I wonder how you managed to run both the original onepagecheckout and the module on the same Magento installation, at the same time?
Thanks
Johan
@Johan — Thanks. We used Google Website Optimizer to setup a simple A/B test. The setup process only took a few hours and we were up and running.
Ok, im using GWO too, soo thats no problem.
What im curious about is how you managed to have both the original and the onepagecheckout side by side och the Magento installation ?
Thanks
Johan
I see. It was because of the changes in the URL path of where the checkout occurs. The OneStepCheckout Magento extension sets it up that way.
In other words, when I setup the A/B test, I could enter the original completely separate URLs:
- ORIGINAL: https://www.youriste.com/checkout/onepage
- Variation 1: https://www.yoursite.com/onestepcheckout
Hey, JR.
How did you configure the “success” page for this test?
When I try to use, for example, /checkout/onepage/success/ , optimizer cannot find that page. When there is nothing in the cart, it redirects to the “/checkout/cart/” page. Of course, Google wouldn’t have anything in the cart.
Any insight would be appreciated!
Mike (or others), did you ever figure out how to set up the success page?
Ok, thanks.
Hi,
Surprising results. What did you use as the success page for GWO?
Mike G
Definitely shocking.
I have been seriously considering this extension.
How many people did you run the test on?
How long did you run the test?
What was your website selling?
Hi,
I think some information from this testing is lacking. There would be lots of readers of this article that would need to know :
What the volume of sales was in the test
Type of demographic
etc
In order to make some sort of judgement. We agree, you must always test which is great.
But what sample size have you used?
Hi Nathan,
I understand the point you’re trying to make but hopefully you understand this is a client test. We can’t give out too much information.
With that being said, if you’ve ever tested with Google Website Optimizer or looked at their test duration calculator, you’ll know that they require a large amount of data in order to find a high confidence winner.
JR…
“…this is a client test. We can’t give out too much information.”
You’re giving out no information.
You could easily post the volume and length of the test, as well as the type of product… cars, flowers, without giving away any client information.
You’re findings are highly suspect, since the One Page Checkout, at least in my mind and everyone I have showed it to, blows away the default magento checkout,.
Sam — Bottom line is, the OneStepCheckout didn’t work with this particular test. The proof is in the pudding.
As I said in that same reply you quoted, if you’ve ever tested with Google Website Optimizer you’ll know that they require a large amount of data in order to find a high confidence winner.
The client we were testing is in the home improvement industry. Each variation had around 125 conversions. That’s pretty substantial evidence and this done with a highly trusted testing solution (Google Website Optimizer).
Let it be known again, as I stated in the post, please don’t jump to a conclusion that you shouldn’t use this extension. In fact, we’re getting ready to launch it again on a few more clients sites. Our goal is to actually show that this extension does in fact, improve conversions.
What findings have you found? Could you share a test with us you’ve run with this extension?
JR – great reminder that what we think should work better doesn’t always do better; no matter how many of our friends and colleagues “THINK” it should do better. I thought the one cart would outperform the multistep process all the time. However, I’ve tested the concept using other ecommerce platforms and have found the same results two different times. That’s the importance of testing as it removes the subjective element and shows you conclusively the reality of what’s better. It sounds like you’re working on other tests and I look forward to updates on how this performs on other Magento based sites. I’ve run several MVT and A/B tests and know it takes a ton traffic for Google Website Optimizer to declare a winner. Without being able to login into your account, I think your screen shot from Google WSO above is plenty to get the point across: STOP GUESSING AND START TESTING!
Haven’t used the Google Website Optimizer yet, but plan to.
I am setting up OneStepCheckOut as we speak, and I’ll let you know how it goes.
I plan on setting it up inline with my shopping cart, since where I really seem to be loosing people is after they’ve added the product to their cart, and before they’re supposed to hit “Proceed to Checkout”