Split test 2 Shopify themes

Highlighted
Shopify Partner
17 0 3

Hello,

I would like to ask if it is possible to A/B split test 2 different Shopify themes?

I have a control theme (https://mystorename.com) and a variation (https://mystorename.com?preview_theme_id=xxxxxx) with a totally different theme from the control in term of design.
Is there a way I can split test the 2 themes?

The problem with appending ?preview_theme_id=xxxxxx is that the user is going to see the theme preview bar at the bottom. 

Even if I hide the bar using .shopify-preview-bar { display: none } the customer will still see it when he reaches checkout.

Do you know how this can be achieved?

Thank you.

1 Like
Shopify Expert
9850 97 1611

A/B testing an entire theme won't be practical - partly for the reason you've noted but mostly since you're throwing too many variables into the mix. A Good A/B test would focus on smaller things so you can understand exactly what might be helping those conversions. Sure it might tell what theme people respond to better (but that's a risky scatter assumption) but you won't know why. The why is important.

I'd also note that unless you're getting decent traffic an A/B test might not yield decent results anyway. If you have small traffic I'd start with having a conversation with some customers first.

★ Winning Partner of the Build a Business competition. ★ http://freakdesign.com.au
0 Likes
Highlighted
Shopify Partner
17 0 3

Thanks Jason.

I have talked to Shopify support shortly after and they said this can not be done at the moment. The only way they advice doing it is simply by publishing the theme and appreciating the results afterwards.

And I do agree to your statement, changes should be made bit by bit. But this is an update of a bigger scale hence the concern.

1 Like
Highlighted
Shopify Partner
2 0 0

One thought I had about this subject is that new themes might be better tested as experimental or beta. Where you ask a number of users to try out the new theme, then try and get their feedback. You can use the qualitative data to help update the new theme and iron out any usability/user experience issues prior to exposing it to your full audience. 

If you hear differing or conflicting feedback from users, it would be a great idea to test those assumptions with a split test. 

The difference being that the new theme is tested on a smaller sample to mitigate risk of negitave impact. Then when you are confident, test to optimize the new theme based on user feedback.

Ideally you want to say most users preferred the new look because they said "x, y, z" and that we tested two variations of a button, for example, and saw a % increase in conversions because of "a". 

Data can give you the evidence you need to be sure about a design solution. But the approach (or experiment) needs to be thought out with some planning.

0 Likes
Highlighted
Explorer
78 2 7

Is it maybe possible now?

0 Likes
Highlighted
New Member
1 0 0

I was able to split test with the instructions from the OP. I didn't see the preview_theme_id line he was concerned about at checkout, and I was able to hide the preview bar according to his instructions.

 

I used Google Optimize to split the traffic evenly between the two themes.

0 Likes