# Run an experiment

> Show some visitors another version of a block or a page, see which one gets more bookings or clicks, and put the winner live through the page's usual review.

Source: https://docs.bifrostorbit.com/cms/editors/experiments/

An experiment, also called an A/B test, shows some visitors a variant of a page, or of one block such as the hero, while the rest see the page as it is. CMS Orbit counts who goes on to do what you want, such as booking a trip, and tells you in plain words which version works better and when the answer is reliable.

For example, Northwilds could test a new photo in the hero of its home page, and count how many visitors go on to pay for a booking.

Your website needs version 0.11 or later of the JavaScript SDK, which runs experiments without further code. Ask your developer to check, and to add the one line that lets visitors take part without cookies (see [How visitors keep their variant](#how-visitors-keep-their-variant)).

## Start an experiment

1. Open the page and select the block you want to try differently, such as the hero.
2. Choose **Test** in the panel on the right, then **Test another hero** (the button is named after the block).
3. Choose what to test, the goal, how many variants and how many visitors take part. Each is described below.
4. Choose **Create and edit the variant**. The editor opens on **Variant B**.
5. Change the variant, then check it with its preview link.
6. Choose **Start**.

**Test** is there for pages with their own address. A page has one experiment at a time: finish the current one before you start another on the same page. Experiments on different pages run side by side.

### What to test

- **One block, such as the hero:** the variant changes that block. The rest of the page is the same for everyone.
- **The whole page:** the variant can change, add, remove or reorder anything on the page.

### What counts as a success

This is the experiment's goal. A visitor counts once, however often they reach it.

| Goal | Counts a visitor who |
| --- | --- |
| A click on a link or button in what you test | Clicks a link or button inside the tested block or page |
| A visit to a page | Opens that page afterwards, such as `/book/thank-you/`. End the address with `*` to count every page under it |
| An event | Does something your website or another Orbit satellite reports, such as **Booking paid** or **Add to bag**. Common events are offered as you type |
| A goal from Insights Orbit | Reaches one of the site's goals. Shown when the space is connected to Insights Orbit |
| A form sent | Sends any form on the website |
| Visitors who stay | Goes on to a second page, clicks anything or stays 30 seconds |

Goals on other pages count too: a visitor who saw the new hero and then pays for the Lofoten ridge walk counts as a booking.

### Variants and visitors taking part

Choose one, two or three variants (A/B, A/B/C or A/B/C/D). Visitors are shared evenly between the original and the variants. **Visitors taking part** sets how many visitors are in the experiment at all; the others see the original and aren't counted.

The box below estimates how long the experiment needs to find a 10% and a 20% improvement, from the page's visitors a day and today's success rate. With Insights Orbit connected, the visitors a day come from the page's last 30 days in Insights Orbit. When the goal is one from Insights Orbit, today's success rate is how often visits through this page reached that goal in the same 30 days. Both are estimates from the past, and you can change them. Fewer visitors, more variants or a smaller share taking part all make it take longer.

### Audience, guardrails and schedule

Open **Audience, guardrails and schedule** for the rest. You can leave all of it as it is.

| Setting | Means |
| --- | --- |
| What you expect | Your hypothesis, such as "A photo of travellers on the trail will get more trip enquiries than the mountain view". It is shown with the results |
| Devices | Computers, tablets and phones. All are ticked to start with |
| Visitors | Everyone, new visitors or returning visitors. New and returning are only known for visitors with the experiment cookie, so others see the original |
| Countries | Two-letter codes, such as `NO, DE, GB`. Empty means everywhere |
| Arriving from a campaign | The campaign, source or medium in the address the visitor arrived on (`utm_campaign`, `utm_source`, `utm_medium`) |
| Guardrails: must not get worse | Other goals to watch, so a win on the goal doesn't cost you elsewhere. **Visitors who stay** is ticked to start with, and you can add one more, such as the event **Booking paid** while the goal is clicks on the hero |
| Sharing the traffic | **An even split**, the clearest answer and best for most tests, or **Move towards the best** (below) |
| Starts, Ends | Leave **Starts** empty to start it yourself once the variant is ready. With **Ends**, it stops by itself |
| At least, days | 7 to start with. A full week evens out weekdays and weekends |

**Move towards the best:** once a day, more visitors are sent to the variant doing best so far, so fewer visitors miss out on it. Every variant keeps at least 10% of the visitors, so a slow starter can still catch up. It suits short campaigns, such as a winter sale, but the answer takes longer to be sure of.

## Edit a variant

The editor opens the variant with a bar under the top bar that names the experiment, switches between **Original** and the variants, and says *Changes here are saved to the variant only. The live page stays as it is.*

- Change only what you want to test. Blocks and fields you leave alone keep following the page, so a typo fixed on the page later is fixed in every variant too.
- In a block test, if you change another block as well, that block becomes part of the test.
- Publishing and deleting aren't on the variant. **Experiment** in the top bar opens the experiment.
- Switch variants after your changes have saved.

To edit the original, edit the page itself.

Once the experiment starts, its variants, goals, audience and traffic are fixed, so the results stay fair. The variant then opens read only. To try something else, stop the experiment and start a new one.

## Check the variants before you start

Each variant has a **Link** in the **Test** panel, and **Preview** and **Copy link** on its card on the experiment's page. Before the start it is a preview of the page's draft that shows that variant, which anyone with the link can open for an hour. While the experiment runs, it is the live page with that variant forced. Visits through these links are never counted.

## Start, pause and stop

- **Start** puts the experiment live. The page must be published first, and each variant must differ from the live page. With a start time in the future, the button says **Schedule**, and it starts by itself.
- **Pause** shows everyone the original and stops counting until you **Resume**. The results so far are kept.
- **Stop** ends the experiment. Everyone sees the original again and no more visitors are counted. The results stay, and you can still choose a winner.

Starting, pausing and stopping reach your website within seconds when it refreshes on webhooks (see [Refresh the website when content changes](https://docs.bifrostorbit.com/cms/developers/webhooks/)).

## Read the results

The **Test** panel shows a summary next to the page. **All the results and settings** opens the experiment's own page, which is also under **Marketing › Experiments**.

The advice at the top says what to do: keep it running and for how long, choose a winner, keep the original, or that there is no real difference. The figures under it are the visitors in the experiment (those who saw the tested part on screen, each counted once), their conversions, the days it has run and roughly how many more days a reliable answer takes at the current traffic. Below them:

- **Variants side by side:** a card for each variant with a live preview of the block being tested (the top of the page for whole-page tests), its rate in large type, and whether it is **Leading**, **Behind for now**, the **Winner** or still **Too early**. **What changed** lists the variant's changes next to the original, with the old and new text and photos. **Preview** opens the variant on the website, and **Edit** or **View** opens it in the editor.
- **How sure are we?:** a curve for each variant showing where its real conversion rate probably lies. Curves far apart mean a clear winner; curves on top of each other mean you can't tell yet. Under 100 visitors per variant, the chart is greyed out and says **Too early**.
- **Better or worse than the original:** for each variant, a dot for the best guess and a bar for the range it is 95% likely to be in. Green means better, red worse, and grey that it could still go either way. Point at a bar for the exact numbers.
- **Progress to a reliable answer:** each variant's visitors against how many it needs, and the days run against the minimum. Until every variant has 100 visitors and the original has a conversion, the bars count towards those first 100.
- **Visitors per day** and **Conversion rate over time:** each variant's visitors day by day, and its rate so far. Lines that cross early and then settle are normal.

**All numbers** opens the full table:

| Figure | Means |
| --- | --- |
| Rate | The share of each variant's visitors who reached the goal |
| Uplift (95% range) | How much better or worse than the original, and the range it is very likely to be in. A range from −5% to +20% means it could still be either |
| Chance to beat the original | How likely the variant is to be better than the original. 50% means you can't tell yet |
| Risk if chosen | How many percentage points of conversion you could expect to lose by choosing this variant if it is in fact worse. The lower the better |

**How to read this** repeats the goal and these explanations.

The advice calls a winner only when every variant has at least 100 visitors and a few conversions, the minimum days have passed, the variant is at least 95% likely to beat the original, and choosing it risks less than 1% of the original's rate. You can look every day: the advice takes that into account.

**Guardrails** show each variant's rate for the goals you asked to protect, as bars. **Getting worse** means a variant is at least 90% likely to make that one worse, and its bar turns red.

**Sample ratio mismatch** is a warning that the variants are getting a different share of visitors than planned, more than chance would explain. Something is wrong, for example a variant that doesn't show for some visitors, so the numbers can't be trusted. Check that every variant shows and works on its preview link.

Visitors who ask not to be tracked, search engines and other bots always see the original and aren't counted.

### With Insights Orbit

When the space is connected to an Insights Orbit site, the experiment's page has **Open in Insights**. It opens the site's experiment report in Insights Orbit, filtered to this experiment from the day it started to today, or to the day it ended. There you can see what else the variants changed: the pages visitors went on to, how long they stayed, the sources they came from and the site's other goals and funnels, for the visits that saw each variant. Insights Orbit only shows these numbers: the experiment is still run, decided and promoted here.

While the experiment is new and its own numbers are too thin to say how long it needs, **Until a reliable answer** is estimated from Insights Orbit: the page's visitors a day in the last 30 days and, for a goal from Insights Orbit, how often visits through the page reached it. The figure is marked **Estimate from Insights Orbit**, and **Progress to a reliable answer** says where the numbers come from. Once every variant has 100 visitors and the original has a conversion, the experiment's own numbers take over.

Insights Orbit also marks on its charts when an experiment starts, pauses, resumes or ends, and when a winner is chosen, with a link back to the results here, and its site overview lists the running experiments. People see the names and results there only if they can see experiments in this space in CMS Orbit.

## Choose the winner

1. Choose **Choose the winner**, in the **Test** panel or on the experiment's page.
2. Pick the variant to keep. The one the advice picked is marked **suggested**. If it looks like it makes a guardrail worse, you are told.
3. Choose **Put Variant B in the draft**, or **Keep the original**.

The winner goes into the page's draft and the experiment ends. From then on every visitor sees the winner, until the page is published. Publish the page, or send it for review, as you would any change: the page's usual review and approvals apply, so nothing goes live without a person approving it. Other unpublished changes in the draft stay and go live with it.

Keeping the original ends the experiment and leaves the page as it is.

## How visitors keep their variant

A visitor keeps the same variant while the experiment runs, so the test is fair. How that works is one setting for the whole website, under **Marketing › Experiments**:

| Choice | Means |
| --- | --- |
| Cookie after consent | The default. Visitors who accept cookies keep their variant for 13 months. Before that, they take part without cookies for the day, if the website passes the daily visitor code |
| Cookieless only | Nothing is stored in the browser. Visitors keep their variant for the day. Needs the daily visitor code from the website, which the SDK adds in one line |
| Cookie without asking | A first-party cookie, `cms_ab`, for 13 months from the first visit. Only where the law allows it without consent, such as outside the EU or in a setup the French data protection authority (CNIL) exempts |

What this means for privacy:

- **Without cookies,** your website's server works out a code for the day from the visitor's browser and their network address with its last part removed, using a secret only the server holds. Nothing is stored on the visitor's device, and the code changes every day. A visitor who comes back another day may see another variant and is counted again.
- **With the cookie,** it holds a random ID, when it was set and the visitor's variant in each experiment. It is used on your website only.
- CMS Orbit receives a different random-looking ID for each visitor in each experiment, so visitors can't be followed from one experiment to another. It stores no address, browser details or cookie.
- Each visitor's record is deleted at the latest 90 days after the experiment ends. The daily totals behind the results stay with the experiment.
- Visitors with **Do Not Track** or **Global Privacy Control** switched on always see the original and aren't counted. So are visitors the website can't place: without the daily code and without the cookie, they see the original.

Which choice fits depends on the law where your visitors are and on your cookie banner. Ask whoever looks after data protection in your organisation.

## Remove an experiment

On the experiment's page, the bin button takes an experiment that isn't running off the list. The page isn't changed. The experiment and its daily results are kept, so the team at Bifröst Orbit can bring it back if it was removed by mistake; the record of which variant each visitor saw is deleted after 90 days.

## Plans and permissions

Experiments are part of the Growth and Enterprise plans. On Starter, **Experiments** and the **Test** panel show a lock with the plan that includes them, and an owner can upgrade under Billing in Orbit.

| Permission | Lets you | Built-in roles |
| --- | --- | --- |
| `experiments:read` | See experiments and their results | Everyone, from Viewer |
| `experiments:manage` | Create experiments, edit their variants, and start, pause and stop them | Editor, SEO Manager, Developer, Administrator, Owner |
| `experiments:promote` | Choose the winner and put it in the page's draft | SEO Manager, Administrator, Owner |

Creating an experiment and editing its variants also need the right to edit the page, in its region. Every experiment created, started, paused, resumed, stopped, promoted or removed, and every change to the setting above, is recorded in the audit log.

AI agents can read experiments and their results, and draft an experiment with its variants for you to check. Only people start, pause, stop and choose winners. See [MCP tools](https://docs.bifrostorbit.com/cms/developers/mcp-tools/).

## In the Orbit app

The space's home in the Orbit app for iPhone shows how many experiments are running. **Experiments** lists them, running and planned first, with each one's visitors, the variant in the lead and the advice. Open one for its results, a chart of the conversion rate over time and any guardrail that looks worse, and to pause, resume or stop it, or choose a winner, with the same permissions as on the web. New experiments are set up on the web.