What's the best method for comparing Shopify theme performance?

Technical Q&A
,

Topic summary

Main issue: How to reliably compare Shopify theme performance after Liquid (Shopify’s templating language) optimizations when results vary across tests.

Observed problem: After publishing a theme and switching to another, initial requests are slower, producing inconsistent Shopify Theme Inspector readings despite visiting key pages (home, catalog, product, contact) 25 times.

Key factors affecting results:

  • Publishing a theme triggers cache-related slowdowns on initial requests.
  • Unpublished themes include redirects and extra overhead (“cruft”).
  • Web performance is asynchronous, especially with JavaScript, introducing variability.

Recommended method:

  • Warm up and measure only after each theme has been live for a short period, ignoring first requests; or
  • Test both themes while unpublished to keep conditions consistent.
  • If the goal is to isolate Liquid performance, disable JavaScript during tests to reduce noise.

Outcome: No single “best” tool or metric established; guidance focuses on test methodology to achieve more stable, comparable results. Discussion remains open for further validation.

Summarized with AI on February 2. AI used: gpt-5.
1

Hello,
I am developing Shopify Theme Optimizer.
I focused on Liquid optimizations.
After optimizing the theme, I compared the themes with the Shopify Theme Inspector, but with very variable results.
I’m wondering if I’ve made a mistake in my comparison method.
I’m comparing it to the water method.
I am publishing the theme that I will profile first.
Then I visit the home page, catalog page, product page and contact page 25 times in a row.
Then I publish the other theme that I will profile.
When I change the published theme, the performance of the first requests is slower than the others.
Is there a right way to compare theme performance?

2

When you publish a theme you will hit cache issues initially as the theme is newly being served.

Unpublished themes have redirects and other cruft added to the them.

Either compare results only from when each theme is live after a short time after making initial requests,

or use results from when they are both unpublished.

Also the nature of the internet is asynchronous, more so when javascript is being used.

So if your focusing only liquid disable javascript.