Welcome to Pack's journey toward enhancing your Hydrogen storefront and shop experience! As part of our commitment to providing a comprehensive digital solution, we are excited to introduce Pack A/B testing. This feature gives you a straightforward method to manage and create content variants for testing purposes across both storefronts and shops.
Got questions?
We would love to help you and hear your thoughts on A/B testing. Join our Discord channel.
Pack’s A/B testing brings the same ease of content management from our customizer into your testing workflow for both storefronts and shops. Additionally, leverage our APIs to ensure flexibility in defining and executing your tests according to your specific needs.
Our team is continuously dedicated to improving our A/B testing product and your experience across all platforms. Keep an eye out for ongoing updates as we introduce new features and optimize your experience.
How to opt-in to Pack A/B Testing
Pack's A/B Testing is currently in opt-in beta, with potential fluctuations in APIs and features ahead. While A/B testing is accessible across all plans during the beta, keep in mind that this could change. Your feedback and insights are invaluable to us as we navigate this phase.
Pack A/B testing is available to everyone as a public opt-in. To opt-in to the beta, follow these steps:
- Go to app.packdigital.com
- Navigate to your storefront or shop
- Go to your Settings found in the left sidebar
- Under Enable Experimental Features, toggle on the A/B Testing (Beta) option
- You should now see A/B Tests in the sidebar. If not, refresh the admin panel.
Please follow the setup requirements below for some additional development steps needed to get A/B testing working in your storefront or shop.
A/B Testing Storefront and Shop Setup Requirements
- A Hydrogen storefront or shop on Pack
- Follow our guide here to add Pack to your existing Hydrogen storefront or shop
- Update Pack packages to A/B testing beta
@pack/hydrogen": "0.2.0-ab-beta.1
- Update your storefront or shop's code to support Pack A/B testing. Follow our guide here.
Managing A/B Tests
In the admin panel, you can view a list of all your A/B tests. These tests can be in a draft, running, paused, or ended state. To view and manage your tests:
- Navigate to your admin panel
- Click A/B Tests in the sidebar
- Create, update, delete, and run tests
You can also delete tests from this view. However, a test needs to be in a paused or ended state before it can be deleted.
Creating a test
- Navigate to your Pack admin panel
- Go to A/B Tests
- Click Create test in the top right
- Enter details about your test:
- Title: The human-readable name for your test
- Test Handle: A unique key for your test, useful for analytics or feature flag testing
- Goal: A high-level description of the overarching objective of your test, grounded in a hypothesis reflecting your desired outcome
- Click Create test in the top right corner to save.
Managing test variants
Once a test is created, you can edit and manage its variants. By default, when a test is created, we provide two variants: a control variant and variant B. These variants have a 50/50 traffic split by default.
When using Pack, you can tweak your test and its variants even while it's running. Just remember, if you make changes to an active test, it's crucial to understand how those changes might impact your test results.
Editing a variant
From the test view, click the action menu on the right side of the variant and select Update variant details.
From here, you can update information about the variant:
- Title: The name for the variant, which can be customized
- Variant Handle: A unique key for the variant, used for analytics or feature flag testing and as the identifier when users are bucketed into a test
- Description: An optional field to describe the intent or changes the variant is supposed to have
- Traffic allocation: Control how traffic is distributed between variants in this test
Adding additional test variants
Pack’s A/B testing allows you to add additional variants to an A/B test, enabling you to run A/B/n tests. To add additional test variants:
- Visit your test
- In the Test Variants table, click Add Variant in the top right
Deleting test variants
If you want to delete a variant from a test:
- Visit your test
- In the Test Variants table
- Click the action menu to the right of the variant you want to delete
- Click Delete variant
- Confirm your variant deletion
If you delete a variant from an active test, users who were bucketed into this variant will be assigned to another variant in the test. Again, it’s crucial to understand how this change might impact your test results before deleting.
Editing content for your test
Once you've defined your test objective and configured your test and its variants, you're ready to edit the content for this test. Start by heading to the Customizer to begin editing your tests.
Enter Test mode
Once you are in the Customizer, enable test mode by clicking the flask icon in the top left corner of the toolbar.
In test mode, you can manage content for each test and its variants. Editing content in test mode assigns changes to the currently active variant. When visitors come to your site and are assigned a variant, they'll see the content saved for that specific test and variant.
Editing a variant’s content
At the top of the toolbar in the customizer, in test mode, select the test and the variant you want to start editing content for.
Changes made to this content will be saved to the variant:
-
Changes on a page level
- Adding new sections
- Removing sections
- Section templates
-
Changes on a global level
- Site settings
- Linked sections or changes to section templates will update wherever they are referenced
When you are outside of test mode, you are editing the control content of that page. Because you can run multiple tests at the same time, it’s important to note that if you edit the control content that is the control in multiple tests while outside of test mode, this may affect your test results.
The customizer will give you a small notice in the top toolbar if content is part of multiple active tests to help prevent unintentional changes during testing.
Starting, pausing, and ending tests
Starting a test
When you’re ready to start your test, you can do it in the admin panel or the customizer.
- To start a test from the admin panel:
- Go to your admin panel
- Select A/B Tests in the sidebar
- Select the test you want to start
- Click Run test in the top right
- To start a test from the customizer:
- Go to your admin panel
- Select Customizer in the sidebar
- Enter test mode
- Select the test you want to start in the toolbar
- Click Run test in the top right
Pausing a test
If there is an issue or an adjustment needed and you need to pause the test, you can do so in the admin panel or customizer.
- To pause a test from the admin panel:
- Go to your admin panel
- Select A/B Tests in the sidebar
- Select the test you want to pause
- Click Pause test in the top right
- To pause a test from the customizer:
- Go to your admin panel
- Select Customizer in the sidebar
- Enter test mode
- Select the test you want to pause in the toolbar
- Click Pause test in the top right
Ending a test
When you want to end a test, you can do so in the admin panel or customizer.
- To end a test from the admin panel:
- Go to your admin panel
- Select A/B Tests in the sidebar
- Select the test you want to end
- Click the action menu
in the top right
- Select End test
- To end a test from the customizer:
- Go to your admin panel
- Select Customizer in the sidebar
- Enter test mode
- Select the test you want to end in the toolbar
- Click the dropdown arrow next to the Pause test button in the top right
- Select End test
Promoting a winner
Once you've completed your test and identified a winning variant, you can promote this successful version to become the new content displayed to your users.
- To promote a test from the admin panel:
- Go to your admin panel
- Select A/B Tests in the sidebar
- Select the test you want to promote
- Click the action menu in the top right
- Select End test
- To promote a test from the customizer:
- Go to your admin panel
- Select Customizer in the sidebar
- Enter test mode
- Select the test you want to promote in the toolbar
- Click the dropdown arrow next to the Pause test button in the top right
- Select End test & promote variant
Manually bucketing yourself into a test
If you want to bucket yourself into a test to validate that everything is working, you can add these query parameters to the URL.
-
- Name
test_id
- Type
- string
- Description
-
A human-readable label for your section.
-
- Name
test_handle
- Type
- string
- Description
-
A unique identifier for your section.
-
- Name
test_variant_id
- Type
- string
- Description
-
An optional string that lets you categorize this section. For example, "Heroes" or "Text Blocks".
-
- Name
test_variant_handle
- Type
- Array<Field>
- Description
-
An array of supported fields that make up your section.
Note: The only requirement is that you include both a test_*
and variant_*
parameter so we know which test and variant to display.
You can use different combinations of the parameters, such as:
?test_handle=ZZZ&test_variant_handle=YYY
?test_id=ZZZ&test_variant_id=YYY
?test_handle=ZZZ&test_variant_id=YYY
?test_id=ZZZ&test_variant_handle=YYY
Resources
Guides
Adding Pack A/B Testing to Your Storefront or Shop
Learn how to add Pack A/B Testing to your existing Hydrogen storefront or shop.