ExperimentationCreating experiments

Randomization units

Overview

This topic explains what randomization units are and how to use them in LaunchDarkly Experimentation. An experiment’s randomization unit is the context kind the experiment uses to assign traffic to each of its variations.

When you create an experiment, you will choose a randomization unit that determines both:

  • which context kind the experiment will allocate traffic to different variations by, and
  • which metrics you can use in the experiment.

Most experiments use the user randomization unit. This means that the experiment will sort contexts that encounter the experiment into different variations by user. You can also choose a different context kind, like device or organization.

Randomization units for metrics and experiments

To use a particular metric within an experiment, the metric and the experiment must use the same context kind as their randomization unit.

You select a metric’s randomization unit when you create it:

The "Metric definition" section of a new metric with the randomization unit called out.

The "Metric definition" section of a new metric with the randomization unit called out.

Later, when you create an experiment, select the same randomization unit to make the metric available to the experiment:

The experiment creation screen with the randomization unit for the experiment called out.

The experiment creation screen with the randomization unit for the experiment called out.

Like experiments, many metrics use a context kind of user as their randomization unit, but can also use other kinds depending on what the metric is measuring. To learn more, read Creating and managing metrics.

Randomization units other than “user”

In LaunchDarkly, the default context kind of user is automatically available for use in Experimentation. If you plan to use a context kind other than user in an experiment, expand the section below to read about the additional steps required.

To use a context kind other than user in an experiment, you must first select an industry-standard randomization unit, and then map that context kind to that randomization unit.

Industry-standard randomization units

There are six industry-standard randomization units that you can use in Experimentation:

  • user
  • user-time
  • guest
  • guest-time
  • organization
  • request

LaunchDarkly limits experiments to randomizing by these six industry-standard units because other units might result in invalid results. If you have a context kind that doesn’t logically map to one of these industry-standard units, it may not be appropriate to use as a randomization unit. For examples of common context kind mapping, read the table in Map randomization units to context kinds.

For example, if you tried to use “country” as a randomization unit, your experiment would serve the same variation to everyone in a particular country. If your experiment had end users in the United States, Mexico, and Canada, the experiment might serve variation A to all end users in the United States and Canada, and variation B to all end users in Mexico. This could result in significantly uneven numbers of end users in each variation, and each variation would not include a random sampling of end users. At the end of the experiment, there would be no way to tell if variation A or B performed better in any given country.

Instead, if you wanted to view results by country, you could filter your results by context attribute. To learn more, read Filtering experiment results.

Map randomization units to context kinds

If you create a context kind other than user that you want to use in experiments, then you must first map the new context kind to one of the industry-standard randomization units. To do this, select Available for experiments when creating a new context kind or editing an existing context kind, and then select the appropriate industry-standard randomization unit. To learn more, read Creating and editing context kinds.

Here is an example of mapping a new context kind of “customer” to the randomization unit “user”:

The "New context kind" dialog with the randomization unit mapping field called out.

The "New context kind" dialog with the randomization unit mapping field called out.

Multiple context kinds can be mapped to the same industry-standard randomization unit.

Examples of common mappings

This table includes the industry-standard randomization units and example context kinds that might be associated with them:

Standard randomization unitExample context kind mappingsDescription
Useruser, member, customerFor each person that encounters your feature, randomly choose a variation for that particular person
User-Timeuser+hour-of-day, user+day-of-week

Takes the form of a composite key of user-key and time-at-some-granularity		For each person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that person/time combination
Guestdevice, cookie, session, logged-out user, guest, non-authorized userFor each non-logged-in person that encounters your feature, randomly choose a variation for that particular guest
Guest-timeguest+hour-of-day, guest+day-of-week

Takes the form of a composite key of guest-key and time-at-some-granularity		For each non-logged-in person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that guest/time combination
Organizationorganization, company, businessFor each organization that encounters your feature, randomly choose a variation for that organization
RequestHTTP request, operation, transaction, interactionFor each particular action that someone performs on your platform, randomly choose a variation for that particular request, operation, or action
Experiments must use valid randomization units

If you’re unsure of which randomization unit to map to your context kind, start a Support ticket for help.

Randomization units and multi-contexts

If you use multi-contexts, you have options as to which context kind you want to use as a randomization unit for an experiment. Expand the section below to view an example.

Imagine the user Anna and the user Jesse both work for the organization Global Health Services. If they are both in an experiment that randomizes by organization, Anna and Jesse will always be sorted into the same variation in the experiment, because their contexts share the same organization. However, if they are in an experiment that randomizes by user, then they could end up in different variations.

Here is what their multi-contexts would look like, though each SDK sends context data to LaunchDarkly in a slightly different format:

Two multi-contexts with the same org key
1{
2  "kind": "multi",
3  "user": {
4    "key": "user-key-anna", // Anna has a unique user key
5    "name": "Anna",
6  },
7  "organization": {
8    "key": "org-key-global-health", // The org key is the same in both multi-contexts
9    "name": "Global Health Services",
10  }
11}
12
13{
14  "kind": "multi",
15  "user": {
16    "key": "user-key-jesse", // Jesse has a unique user key
17    "name": "Jesse",
18  },
19  "organization": {
20    "key": "org-key-global-health", // The org key is the same in both multi-contexts
21    "name": "Global Health Services",
22  }
23}

There are two ways you could randomize the multi-contexts in an experiment:

  • if you randomize by user, Anna could be assigned to one variation, and Jesse could be assigned to the other variation because they have different user keys
  • if you randomize by organization, Anna and Jesse will both always be assigned to the same variation because they share the same organization key