A/B Testing (Beta)

 

Overview

Welcome to Pack's journey toward enhancing your Hydrogen storefront experience! In line with our commitment to providing a comprehensive digital solution, we're thrilled to introduce Pack A/B testing. This feature empowers you with a straightforward method to manage and create content variants for testing purposes.

Pack’s A/B testing brings the same ease of content management from our customizer into your testing workflow. Additionally, leverage our APIs to ensure flexibility in defining and executing your tests according to your specific requirements.

Our team is continuously dedicated to enhancing our A/B testing product and your experience. Keep an eye out and expect ongoing updates as we introduce additional features and optimize experiences.

How to opt-in to Pack A/B Testing

Pack A/B testing is a public opt-in. To opt-in to the beta, please follow the steps below:

1. Go to app.packdigital.com
2. Navigate to your storefront
3. Navigate to your Settings found in the left sidebar
4. Under Enable Experimental Features enable the A/B Testing (Beta) flag
5. You should now see A/B Tests on the sidebar, if not, give the admin a refresh

Follow the storefront requirements below for some small additional storefront development set up to get A/B testing working.

A/B Testing Storefront Setup Requirements

1. A Hydrogen storefront on Pack
   • Follow our guide here to add Pack to your existing Hydrogen storefront
2. Update Pack packages to A/B testing beta
   • @pack/hydrogen": "0.2.0-ab-beta.1
3. Update your storefront's code to support Pack A/B testing, follow our guide here

Managing A/B Tests

In the admin, you can see a list of all your A/B tests. These tests can be in a draft, running, paused, or ended state. To view and manage your tests:

1. Navigate to your admin
2. Click A/B Tests in the sidebar
3. Create, update, delete, and run tests

From here, you can delete tests from this view too. A test needs to be in a paused or ended state to be deleted.

Creating a test

1. Navigate to your Pack admin
2. Go to A/B Tests
3. Click Create test in the top right
4. From here, you will be able to enter details about your test.
   • Title: The title of the test you want to create. This is a human-readable name for your test.
   • Test Handle: A unique key for this test. This key can be used for your analytics or feature flag testing.
   • Goal: This is a high-level description of the overarching objective of your test, grounded in a hypothesis reflecting your desired outcome
5. You can then click Create test in the top right corner to save

Managing test variants

Once a test is created, you can edit and manage your variants. By default when a test is created, we provide two variants, a control variant and your second variant-b. Both these variants by default have their traffic split 50/50.

Editing a variant

 

From the test, you can click the action menu on the right of the variant and Update variant details.

From here you can update information about the variant

1. Title: This is the name for the variant, and can be name to whatever you like
2. Variant Handle: Is a unique key for the variant. Like the Test Handle, this variant handle can be used for your analytics, feature flag testing, and will be the identifier when users are bucketed into a test.
3. Description: This is an optional, but can be helpful to describe the intent or changes that this variant is supposed to have.
4. Traffic allocation: This is how you can control the distribution of traffic that would get bucket into this variant for this test.

Adding additional test variants

Pack’s A/B testing allows you to add additional variants to an A/B test so you can run A/B/n tests. To add additional test variants:

1. Visit your test
2. In the table of Test Variants click Add Variant in the top right

Deleting test variants

If you would like to delete a variant on a test:

1. Visit your test
2. In the table of Test Variants
3. Click the action menu to the right of the variant you’d like to delete
4. Click Delete variant
5. Confirm your variant deletion

If you do delete a variant on an active test, users who were bucketed in this variant will be assigned another variant in this test. Again, it crucial to understand how this change might impact your test results before deleting..

Editing content for your test

Once you've defined your test objective and configured your test and its variants, you're ready to edit content for this test. Begin by heading the Customizer to start the editing your tests.

Enter Test mode

 

Once you are in the Customizer, enable test mode by clicking the flask in the top left corner of the toolbar.

In test mode, you can manage content for each test and its variants. Editing content in test mode assigns changes to the currently active variant. When visitors come to your site and are assigned a variant, they'll see the content saved for that specific test and variant.

Editing a variant’s content

At the top of the toolbar in the customizer, in test mode, you can select the test and the variant that you want to start editing content for.

Changes made to these content will be saved to the variant:

• Changes on a page level
  • Adding new sections
  • Removing sections
  • Section templates
• Changes on a global level
  • Site settings
  • Linked sections or changes to section templates will update wherever they are referenced

When you are outside of test mode, you are editing the control content of that page. Because you are able to run multiple tests at the same time, it is important to note, if you are outside of test mode and editing the control content that is the control on multiple tests, this may affect your test results.

The customizer will give you a small notice in the top toolbar if content that is part of multiple active tests to help prevent unintential changes mid testing.

Starting, pausing, and ending tests

Starting a test

When you are ready to start your test, you can do it in the admin or in the customizer.

• To start a test from the admin:
  • Go to your admin
  • Select A/B Tests in the sidebar
  • Select the test you want to start
  • At the top right, you can click Run test
• To start a test from the customizer:
  • Go to your admin
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to start in the toolbar
  • At the top right, click Run test

Pausing a test

If there is an issue or an adjustment needed to be made and you need to pause the test. You can pause the test in the admin or customizer.

• To pause a test from the admin:
  • Go to your admin
  • Select A/B Tests in the sidebar
  • Select the test you want to pause
  • At the top right, you can click Pause test
• To pause a test from the customizer:
  • Go to your admin
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to pause in the toolbar
  • At the top right, click Pause test

Ending a test

When you want to end a test, you can do so in the admin or customizer.

• To end a test from the admin:
  • Go to your admin
  • Select A/B Tests in the sidebar
  • Select the test you want to end
  • At the top right, click the action menu
  • Select End test
• To end a test from the customizer:
  • Go to your admin
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to end in the toolbar
  • At the top right, click the more arrow dropdown on the Pause test button
  • Select End test

Promoting a winner

Once you've completed your test and identified a winning variant, you can promote this successful version to become the new content displayed to your users.

• To promote a test from the admin:
  • Go to your admin
  • Select A/B Tests in the sidebar
  • Select the test you want to promote
  • At the top right, click the action menu
  • Select End test
• To promote a test from the customizer:
  • Go to your admin
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to promote in the toolbar
  • At the top right, click the more arrow dropdown on the Pause test button
  • Select End test & promote variant

Manually bucketing yourself into a test

If you would like to bucket yourself into a test to validate that everything is working, you can add these query parameters to the URL.

Note, the only requirement is that you have both a test_* and variant_* parameter included so we know what variant and test to display.

It is possible to use different combinations of the parameters, such as:

• ?test_handle=ZZZ&test_variant_handle=YYY
• ?test_id=ZZZ&test_variant_id=YYY
• ?test_handle=ZZZ&test_variant_id=YYY
• ?test_id=ZZZ&test_variant_handle=YYY