AB Testing Apps

Highlighted
New Member
1 0 0

I'm looking to AB test a new app on the product page (a pop up appears on the product page offering a bundle deal). I want to see the impact of this app on sales, benchmarked against the product page's performance without this app. My sales are pretty volatile, so it's not possible to observe this impact by looking at historical sales.

I tried using google analytics experiments to do this, but it requires a code snippet to be placed on the specifc pages where the experiment is running, but Shopify does not allow that.

Any advice how to do run AB experiments on Shopify to test the impact of an app?

0 Likes
Highlighted
Explorer
57 0 18

What about just trying one app for some time, check the results and then try another app for some time?
I know it is not the best way, but if you want to really AB test your apps, then you will need some code which will turn an app on for some users and off for others. I don't know if you already have this implemented, but if you have this, then you are not far away from implementing the tracking mechanism.
 

But you will definitely need to edit the liquid files.

Adele
0 Likes
Highlighted
Shopify Partner
2 0 0

For the technical hurdle:

If you use Google Tag Manager (GTM) you can deploy Google Analytics code and snippets based on page URL. The code is injected to the page via what they call a Trigger. The Triggers can be based on Host, URL, etc. 

You can install GTM on all pages of your site, then choose what tags (code snippets) should be fired where/when (trigger configs). 

I use it alot when A/B/n testing new pages or sections of a site. Google generally has good documentation and most things are free to use.

*HOWEVER*

Successful A/B or multivariate tests isolate a variable and test the amount of change. When you test a new theme, which may be very different from the existing design, would mean trying to isolate many variables. Practically speaking, this could take a very long time - lots of complexity.

A/B is great for optimization purposes. If you are introducing a new theme, then you should talk to users and work out usability or user experience issues before go-live. Then after you launch you can optimize the new theme based on the data you see.

In both cases though, you'll want solid tracking. None of the cases would mean much if your instrumentation isn't installed and configured properly. 

0 Likes
Highlighted
Shopify Partner
17 1 3

Hi Docstore, you may want to consider using Optimizely for this: https://apps.shopify.com/optimizely-1

Unfortunately it may be difficult to test this specific change, since most Shopify apps are loaded asyncronously, and Optimizely is loaded early in the page lifecycle.

But Optimizely is pretty robust in my experience, there may be an option to poll for the presence of a DOM element, which you can then hide, for example.

Hope that helps!

0 Likes
Highlighted
Shopify Expert
714 54 160

For anyone wanting to A/B test apps, you'll need to swap themes or hide the app code.

 

For isolating apps there are 2 options:
a) The easy way is commenting out the app snippet - usually found in theme.liquid or the specific place it shows (product.liquid maybe if it's product page only).

 

b) If the app injects the script on the backend, rather than in your theme code, you can use this tactic to prevent the script from loading when Shopify calls it's script injection:

<!-- First capture the content for header object -->
{% capture header_content %}
  {{ content_for_header }}
{% endcapture %}

<!-- Then remove the url from that content -->
{{ header_content | remove: 'url of script injected' }} 

To get the url of the script injected, you can "view source" on your page in your browser.

To sum up the testing:

1) Duplicate your live theme.

2) Hide the app snippet or script in the new theme.

3) Swap the live theme and duplicate theme (I prefer daily at midnight).

4) Record the analytics and compare your stats.

 

To save you time and get more accurate results, you can use the Theme Scientist A/B testing app that my team built. I use this on a regular basis on Shopify sites so if you need any tips give me a shout.

 

 

 

 

 

Founder of Speed Boostr (Shopify optimization and dev agency).
Creator of Theme Scientist (A/B testing app).
Creator of Order Automator (app that auto tags + fulfills orders, send emails).
Creator of Tip Jar (add a tip button to your store).
Creator of Shopify Analyzer (free performance analysis tool).
My team also builds custom apps and automated solutions.
Enjoy the adventure!
0 Likes