I try to do A/B testing as much as possible, but sometimes I feel I could be over testing.
For example I am testing some Klaviyo sign-up forms, and often I will have four variations running on 1 form. I might test whether it triggers based on exit event or 20 seconds versus 30 seconds, how many items they have in their cart or just trigger it on the cart or check out page. Then I also have different color schemes for the sign-up forms and of course different text content variations.
Should one test just element or variable at a time to be sure that it was a statistically significant determining factor or what is the best practice?
Great questions, Pepper_Pantry.
It’s great that you’re committed to testing, but more tests don’t always mean better insights. The key is to prioritize what will have the biggest impact while keeping some level of guardrails to ensure clean data.
It’s okay to test multiple elements at once, but be mindful of keeping them isolated to some extent to avoid polluting your results. If you change too many things simultaneously, it becomes difficult to pinpoint what’s driving the outcome.
A good approach is to run these tests in parallel but stagger them over time. For example, launch just one variant per week instead of all at once. This helps maintain clarity, making it easier to interpret the results and draw actionable conclusions.
Hope this helps!
Carlos
Visit our website
Follow me on LinkedIn