Hey @ivy_help , here's how I would approach this:
1) Ask the app developers if you can load their app via snippet. Some apps load with a theme snippet, while some inject the scripts into the live site.
Even apps that don't have a theme snippet, I've had success converting to that load style just by asking them.
2) Now you have the snippet that controls whether the app loads, so duplicate your live theme, then remove the snippet from this duplicate theme.
3) Now rotate the 2 themes every day at the same time, so each theme / app gets the same exposure.
4) For best results make the change at exactly 00:01 in your local store time. This will give you the cleanest analytics.
If you want this automated, check out Theme Scientist A/B testing app. I built this as a way to run quality A/B tests on autopilot, then turned it into an app for everyone.
I know it's been a few months but if you still have ideas and need help setting up the app test let me know. I don't think I've A/B tested 2 different apps before, I like the concept though and would happy to help get you set up.
Founder of
Speed Boostr (Shopify optimization experts, theme customization, custom app development).
Creator of
Shopify Analyzer (first performance analysis tool for Shopify sites, free for the Shopify community).
Creator of
Order Automator (auto tag orders and customers + auto fulfill orders + email notifications).
Creator of
Theme Scientist (A/B testing app) and
Tip Jar (add a tip button to your store).
