What you can and cannot test with Shopify Rollouts

Rollouts tests whole themes and whole checkout configurations. It doesn't test individual elements, and it isn't a price testing tool. Knowing which side of that line your idea falls on saves you from designing a test the platform can't run.

This post goes into the detail. If you're still deciding whether Rollouts can replace your CRO app, start with our overview post instead.

What is the unit of a test?

A whole theme, or a whole checkout and account configuration. Not a section, a block or a single button.

When you start a rollout, Shopify duplicates your published theme, and the variant is that copy with your changes applied. Each visitor is assigned to one version and stays on it. Because that happens server-side before the page renders, there's no flicker and no measurable hit to speed.

The catch is that everything you change goes into the same variant. Move the reviews block, rewrite the delivery copy and swap the gallery layout, and you get one result covering all three. It'll tell you which version won, but not which change made the difference.

Launch or experiment: which do I want?

An experiment splits traffic and measures the difference. Use it when you have a hypothesis and want an answer.

A launch goes out to everyone with no split, but in stages, with a rollback option. Use it when you've already made the decision and want to limit the damage if something breaks, like a replatformed theme, a seasonal redesign, or anything where the risk is a bug rather than a wrong hypothesis.

People tend to pick experiment by default when what they want is a launch. If the result isn't going to change what you do, you don't need the split, and a launch gets you there faster.

Keep in mind the plan limits: experiments need Grow or above. Launches, scheduling and rollback are available on lower plans.

What can I test?

Anything you can express as a theme change or a checkout configuration:

• Product page structure, like the order of the gallery, buy box, reviews and details
• Homepage layout and what appears above the fold
• Navigation and category structure
• Cart behavior, including a cart drawer versus a full cart page
• Checkout configuration and the account experience
• A complete redesign against your current theme

That last one is where Rollouts is strongest and third-party tools are weakest. Testing a full redesign client-side means rewriting most of the page after it loads, which is exactly when flicker gets unbearable.

What can I not test?

Individual elements on their own. You can change just one thing, but you're still shipping a whole copy of the theme to do it. That works, it's just a lot of effort for a button.

Price. Prices live on products, not in the theme, so both variants use the same ones. You still need a purpose-built tool for price and offer testing.

Anything outside the storefront. Email, ads and post-purchase flows aren't part of your theme.

Personalization by segment. Rollouts splits traffic to measure a difference. It isn't a targeting engine for permanently showing different experiences to different audiences.

How should I design a test given all that?

Three habits make whole-theme testing work.

Write the hypothesis first, in one sentence, with a number. "Moving reviews above the fold on product pages will raise add-to-cart rate" is testable. "Improving the product page" isn't. The tool won't break a multi-change variant down for you, so your hypothesis is the only thing that lets you make sense of the result. If you're not sure what to test yet, a CRO audit will show you where the biggest drop is.

Group changes that share a reason. If your hypothesis is that shoppers don't feel confident at the buy box, then reviews, returns copy and delivery estimates all belong in the same variant, because they're one idea expressed three ways. Change those and the homepage hero as well, and you've got two ideas you won't be able to separate.

Decide when you'll stop before you start. Settle how long it runs and what result would change what you do. Whole-theme tests produce bigger effects than element tests, so they reach significance sooner, which makes it tempting to peek and stop early on what's really just noise.

Where does it leave third-party tools?

With a narrower job, mostly covering price, offers and heavy segment-level personalization. For testing the structure of your storefront, the native tool is now the better option. There's no flicker, no extra scripts, no monthly fee, and the test can't drift because Shopify takes a snapshot of the theme when the rollout starts.

We plan and run whole-theme tests for Shopify brands. The hypothesis is the hard part, not the tooling.