Shopify Rollouts: do you still need a CRO app?

Shopify added native A/B testing to Rollouts on June 5, 2026. It's free, it runs server-side, and you don't need an app. It also tests whole themes rather than individual elements, which makes it a different kind of tool from Intelligems or Convert, not just a cheaper version of one.

Whether you can cancel your CRO app comes down to what you were testing with it. For a lot of brands, the answer is yes.

What is Rollouts?

It's a way to change your storefront gradually instead of all at once. Rollouts come in two types, and only one of them is a test.

A launch pushes a change to all eligible visitors with no split. You get scheduling and a rollback button, which makes it a deployment tool rather than an experiment.

An experiment splits traffic between your published experience and a variant, then reports which one did better. That's your A/B test.

Both work at the level of a whole theme, or a whole checkout and account configuration. When you start one, Shopify copies your published theme so you can keep editing the live store while the test runs. That gets rid of the most common way in-house tests get contaminated.

What does it cost?

Nothing on top of your existing plan, but experiments are plan-gated. A/B experiments need Grow or above. On Basic, you only get scheduling and rollback.

That's worth running the numbers on. If you're on Basic and paying for a CRO app just to run tests, upgrading to Grow might cost less than the app. And if you're on Plus, you already have it.

Is server-side testing better?

Yes, and it's the best argument for Rollouts over most third-party tools.

Client-side testing works by loading JavaScript, letting it pick which variant to show, and then rewriting the page. That causes flicker, where the original page shows for a moment before it gets replaced. Flicker looks bad, but it also skews your results, because the variant with the visible flash converts worse partly because of the flash.

It slows down every page load for every visitor too, including people who aren't in a test. Shopify's testing happens before the page reaches the browser, so you don't get either problem.

So do I still need a CRO app?

Look at what you ran last year.

You probably don't need one if your tests were whole-page or whole-theme redesigns, new homepage layouts, restructured navigation or checkout configuration changes. Rollouts covers all of these, and does them better than a client-side tool.

You probably still do if you test individual elements, like button copy, a single product-page module or a badge, or if you run price and offer tests. Price testing especially isn't what Rollouts is built for, so you'll still need a third-party tool for that.

For most of the brands we work with, Rollouts covers the majority of tests worth running, and the ones it doesn't cover tend to have the smallest effects anyway. A button color test needs huge amounts of traffic to reach significance and rarely moves the number. A restructured product page does.

What changes about how I plan tests?

Testing whole themes changes the unit of work. Instead of shipping one isolated tweak, you're building a complete alternative version of your storefront.

That sounds like more work, but mostly it isn't, because testing single elements never really reflected how websites work. A better product page usually needs the navigation, trust signals and imagery to change together. Testing them one at a time measures each change in a page that was designed around the other version.

The trade-off is fewer, bigger tests. You'll run fewer of them and learn more from each one, as long as you can still explain why the winner won. That means writing down a clear hypothesis before you start, because the tool won't break the result down for you.

What should I do about it?

If you have a CRO app, look back at the last twelve months of tests and count how many were whole-page. If most of them were, run your next test in Rollouts and compare the experience before you renew.

If you've never tested because the tools were too expensive, that excuse is gone. Start with the page that gets the most traffic and has the weakest conversion rate, which is usually a product page or the cart.

We run CRO for Shopify brands and have moved most of our whole-page testing to Rollouts. Our next post goes into detail on what it can and can't test.