Split test 2 Shopify themes

[MADahmani]

Hello,

I would like to ask if it is possible to A/B split test 2 different Shopify themes?

I have a control theme (https://mystorename.com) and a variation (https://mystorename.com?preview_theme_id=xxxxxx) with a totally different theme from the control in term of design.
Is there a way I can split test the 2 themes?

The problem with appending ?preview_theme_id=xxxxxx is that the user is going to see the theme preview bar at the bottom. 

Even if I hide the bar using .shopify-preview-bar { display: none } the customer will still see it when he reaches checkout.

Do you know how this can be achieved?

Thank you.

[Jason]

A/B testing an entire theme won't be practical - partly for the reason you've noted but mostly since you're throwing too many variables into the mix. A Good A/B test would focus on smaller things so you can understand exactly what might be helping those conversions. Sure it might tell what theme people respond to better (but that's a risky scatter assumption) but you won't know why. The why is important.

I'd also note that unless you're getting decent traffic an A/B test might not yield decent results anyway. If you have small traffic I'd start with having a conversation with some customers first.

[MADahmani]

Thanks Jason.

I have talked to Shopify support shortly after and they said this can not be done at the moment. The only way they advice doing it is simply by publishing the theme and appreciating the results afterwards.

And I do agree to your statement, changes should be made bit by bit. But this is an update of a bigger scale hence the concern.

[Ken16]

One thought I had about this subject is that new themes might be better tested as experimental or beta. Where you ask a number of users to try out the new theme, then try and get their feedback. You can use the qualitative data to help update the new theme and iron out any usability/user experience issues prior to exposing it to your full audience. 

If you hear differing or conflicting feedback from users, it would be a great idea to test those assumptions with a split test. 

The difference being that the new theme is tested on a smaller sample to mitigate risk of negitave impact. Then when you are confident, test to optimize the new theme based on user feedback.

Ideally you want to say most users preferred the new look because they said "x, y, z" and that we tested two variations of a button, for example, and saw a % increase in conversions because of "a". 

Data can give you the evidence you need to be sure about a design solution. But the approach (or experiment) needs to be thought out with some planning.

[ilja]

Is it maybe possible now?

[Pierre-US]

I was able to split test with the instructions from the OP. I didn't see the preview_theme_id line he was concerned about at checkout, and I was able to hide the preview bar according to his instructions.

I used Google Optimize to split the traffic evenly between the two themes.