AB Testing
Overview
Introduction
A/B Testing allows advertisers to segment the users they’re reaching on X so that they can understand how best to optimize for campaign performance and gather learnings to inform their marketing strategies.
These segments—referred to as user group splits—are randomized and mutually exclusive. With randomization, factors that influence outcomes are equally distributed. In other words, there are no inherent differences between groups or their expected behaviors. Because of this, when a single variation is applied to a user group and not the others, the difference in campaign performance can be attributed to that variation.
While it’s possible to test many variations at once, we strongly recommend testing a single variation at a time. This isolates the causal factor for the observed difference in campaign performance.
Variations are set at the campaign level. For example, if the advertiser wants to test the efficacy of a new creative, they should create two identical campaigns where the only difference is the creative. In the future, we plan to support variations at the line item level.
Use Cases
A/B testing is most often used to support (1) optimization use cases for performance customers who want to understand what works best on X in order to optimize their investment and (2) learning use cases for brand advertisers who want to use learnings to inform their marketing strategy.
The API will support A/B testing for any campaign variable, including:
-
Creative
-
Targeting
-
Bid type
-
Bid unit
A/B Testing
A/B Testing allows advertisers to segment the users they’re reaching on Twitter so that they can understand how best to optimize for campaign performance and gather learnings to inform their marketing strategies.
These segments—referred to as user group splits—are randomized and mutually exclusive. With randomization, factors that influence outcomes are equally distributed. In other words, there are no inherent differences between groups or their expected behaviors. Because of this, when a single variation is applied to a user group and not the others, the difference in campaign performance can be attributed to that variation.
While it’s possible to test many variations at once, we strongly recommend testing a single variation at a time. This isolates the causal factor for the observed difference in campaign performance.
Variations are set at either the campaign level or the ad group level. The ad group is set through line item in Ads API, As an example of an ad group level variation, if the advertiser wants to test the efficacy of a new creative, they should create one campaign with 2 identical ad groups where the only difference is the creative.
Use Cases
A/B testing is most often used to support (1) optimization use cases for performance customers who want to understand what works best on Twitter in order to optimize their investment and (2) learning use cases for brand advertisers who want to use learnings to inform their marketing strategy.
The API will support A/B testing for any campaign variable, including:
-
Creative
-
Targeting
-
Bid type
-
Bid unit
Attributes
A/B Tests are represented as nested structures. There are top level fields for the A/B Test itself and an array of user group objects, each with a set of fields describing it.
At a high level, every A/B Test must include the following information.
-
The test duration, represented by the start_time and end_time fields
-
The level at which the split will occur, represented by the entity_type field
-
At least two (and at most 30) user groups, each represented as an object in the user_groups array
Each user group must include the following information.
-
The percentage of users that should be allocated to the given user group, represented by the size field
-
The campaign IDs/line item IDs that should make up the pool of users for the given user group, represented by the entity_ids array
Optionally, name and description values can be set for A/B Tests and for user groups. Information about validation rules and other constraints can be found below.
Other metadata, such as the ID or the created at time, are also included, but are automatically set by Twitter.
An example A/B Test entity for campaign level is shown below.
An example A/B Test entity for line item level is shown below.
Usage
The subsections below describe creating and updating A/B Tests. Reading and deleting work like they do with all other Ads API endpoints.
Creating
Create an A/B Test using the POST accounts/:account_id/ab_tests endpoint. The endpoint only accepts JSON POST bodies. The Content-Type must be set to application/json.
After the advertiser sets up two or more campaigns, an A/B Test can be created. As stated above, A/B Tests must include: test duration, split level, and at least two user groups. Each user group must declare the percentage of users that should be allocated to it as well as the campaign IDs that should make up its pool of users. Each of these is described in further detail below.
Test duration:
-
The start_time and end_time values must
-
Be in the future (relative to when the A/B Test is created)
-
Overlap with the campaign/line item flight dates
-
-
The test must last at least one day for non-app-based campaigns and last at least five days for app-based campaigns
Split level:
- The entity_type can be set to CAMPAIGN or LINE_ITEM
User groups:
-
Each user group is represented as an object in the user_groups array
-
A minimum of two user groups is required
-
A maximum of 30 user groups is allowed
-
-
The size for each user group is set using a string representation of a numeric value between 1.00 and 99.00
- Note: The size values across objects must add up to 100.00
-
The campaign IDs should be specified in each user group’s entity_ids array
Optionally, set the name and description for the A/B Test or for one of more user groups.
The following request creates an A/B Test on the campaign level that lasts four days and has two user groups with 50% of users in each group. The first user group is based on campaigns f2qcw and f2tht; the second user group is based on campaigns f2rqi and f2tws. The request also adds names and descriptions to some parts of the entity.
twurl -X POST -H ads-api.x.com “/8/accounts/18ce54d4x5t/ab_tests” -d ’{“end_time”: “2020-12-05T01:00:00Z”, “entity_type” : “CAMPAIGN”, “start_time”: “2020-12-01T01:00:00Z”, “user_groups”: [{“entity_ids”: [“f2qcw”, “f2tht”], “size”: “50.00”, “name”: “first group”},{“entity_ids”: [“f2rqi”, “f2tws”], “size”: “50.00”, “name”: “second group”, “description”: “second AB test group”}], “name”: “first AB test”, “description”: “documentation example”}‘
For A/B Testing on line item level
The major difference between A/B testing on the campaign and the line item levels is the entity_type. We need to set it to ‘entity_type’ = ‘LINE_ITEM’ for A/B tests on the line item level. This applies to all the actions on an already created A/B test below.
Requirements:
- All line items of the A/B testing campaign must be included in the split test.
- Only equal splitting is allowed for Line item level.
- Number of user groups line items allowed in 1 split test must be less than or equal to 5.
- Only 1 line item per user group.
Updating
Update an A/B Test using the PUT accounts/:account_id/ab_tests/:ab_test_id endpoint. The endpoint requires that a JSON blob be sent in the request. The Content-Type must be set to application/json.
Like other update endpoints, the PUT accounts/:account_id/ab_tests/:ab_test_id endpoint requires that the A/B Test ID be referenced in the URL. In general, A/B Tests can only be updated while the status is SCHEDULED. There is one exception: it’s possible to update the A/B Test’s end_time while it’s LIVE.
This endpoint supports partial JSON with object IDs. The following principles apply:
-
To add or remove objects or elements, pass in the entire array (and its substructures); this is a replacement operation
-
Otherwise, modify (change, add, remove) existing fields by referencing key names or IDs
-
To remove a field, set its value to null
-
Fields that are not passed in are not modified
-
For example, adding a third user group to the previously created A/B Test would require us to send the user_groups array with the two existing user group objects as well as with the new one we’d like to add. Think of this as recreating the user_groups array; pass the data in as if you were creating it this way to begin with (do not pass in user group object IDs). The user_groups array in the update request could be represented as follows.
Notice that the size values across objects still add up to 100.00. If we would not have updated them for the first two objects—previously set at 50.00 each—the request would have failed.
If, instead, we simply wanted to add a description to the first user group, the user_groups array in the update request would be represented as follows.
We reference the user group object by ID and only include the field we’d like to modify.
Request Examples
This section provides additional example update requests. Think of these as being called sequentially. JSON blobs are formatted for readability. Responses are omitted.
To make the following modifications, the request would be represented as follows. (This is the same as the previously used example above.)
-
Adds a third user group without a name or description
-
Changes the percentage of users in each user group
twurl -X PUT -H ads-api.x.com “/8/accounts/18ce54d4x5t/ab_tests/hr7l0” -d ‘
To make the following modifications, the request would be represented as follows.
-
Removes the A/B Test description
-
Adds a description to the first user group
-
Add an entity ID (f2syz) to the second user group
twurl -X PUT -H ads-api.x.com “/8/accounts/18ce54d4x5t/ab_tests/hr7l0” -d ‘
The third modification requires that we pass in the two existing entity IDs along with the new one. Notice that no changes were made to the third user group.
To make the following modifications, the request would be represented as follows.
-
Removes the second user group
-
Changes the percentage of users in each user group
API reference
AB Tests
GET accounts/:account_id/ab_tests
Retrieve details for some or all A/B tests.
Resource URL
https://ads-api.x.com/12/accounts/:account_id/ab_tests
Parameters
Name | Description |
---|---|
account_id required | The identifier for the leveraged account. Appears within the resource’s path and is generally a required parameter for all Advertiser API requests excluding GET accounts. The specified account must be associated with the authenticated user. Type: string Example: 18ce54d4x5t |
ab_test_ids optional | Scope the response to just the desired A/B Tests by specifying a comma-separated list of identifiers. Up to 200 IDs may be provided. Type: string Example: hr7l0 |
count optional | Specifies the number of records to try and retrieve per distinct request. Type: int Default: 200 Min, Max: 1 , 1000 |
cursor optional | Specifies a cursor to get the next page of results. See Pagination for more information. Type: string Example: 8x7v00oow |
live_during optional | Scope the response to A/B Tests that were or that will be live during the given date range. This returns A/B Tests whose start and end times overlap—partially or fully—with the given date range. Specify values as comma-separated dates, expressed in ISO 8601. The earlier date should be specified first. Type: string Example: 2020-11-01T08:00:00Z,2020-12-01T08:00:00Z |
q optional | An optional query to scope resource by name . Omit this parameter to retrieve all.Type: string Min, Max length: 1 , 80 |
sort_by optional | Sorts by supported attribute in ascending or descending order. See Sorting for more information. Type: string Example: created_at-asc |
status optional | Scope the response to A/B Tests with the desired status. Type: enum Possible values: COMPLETED , LIVE , SCHEDULED |
user_id optional | Scope the response to A/B Tests created by the specified user ID. Note: Cannot be specified at the same time as username .Type: long Example: `756201191646691328. |
username optional | Scope the response to A/B Tests created by the specified username. Do not include the leading ”@” symbol. Note: Cannot be specified at the same time as user_id .Type: string Example: `apimctestface. |
with_deleted optional | Include deleted results in your request. Type: boolean Default: false Possible values: true , false |
Example Request
GET https://ads-api.x.com/12/accounts/18ce54d4x5t/ab_tests
Example Response
POST accounts/:account_id/ab_tests
Create a new A/B Test.
All parameters are sent in the request body and a Content-Type
of application/json
is required.
Resource URL
https://ads-api.x.com/12/accounts/:account_id/ab_tests
Parameters
Name | Description |
---|---|
account_id required | The identifier for the leveraged account. Appears within the resource’s path and is generally a required parameter for all Advertiser API requests excluding GET accounts. The specified account must be associated with the authenticated user. Type: string Example: 18ce54d4x5t |
end_time required | The time, expressed in ISO 8601, that the A/B Test will end. Type: string Example: 2020-10-02T00:00:00Z |
entity_type required | The type of entity to be used for user group splits. Type: enum Possible values: CAMPAIGN , LINE_ITEM |
start_time required | The time, expressed in ISO 8601, that the A/B Test will begin. Type: string Example: 2022-05-30T00:00:00Z |
user_groups required | Describes the user groups. More information in the table below. Between 2 and 30 user groups may be specified. Type: array of objects |
description optional | The description for the A/B Test. Maximum length: 1,024 characters. Type: string Example: documentation example |
name optional | The name for the A/B Test. Maximum length: 255 characters. Type: string Example: first AB test |
User Groups
Name | Description |
---|---|
entity_ids required | An array of entity IDs. Note: Entities can only be associated with one A/B Test. Type: array Example: ["dxi0l", "e66bl"] |
size required | The percentage of users to allocate to this user group. This is a numeric value represented as a string with at most two digits after the decimal point. For example, represent 40% as either: 40, 40.0, or 40.00. Note: The size values across objects must add up to 100.00. Type: array Min, Max: 1.00 , 99.00 |
description optional | The description for the user group. Maximum length: 1,024 characters. Type: string Example: second AB test group |
name optional | The name for the user group. Maximum length: 255 characters. Type: string Example: first group |
Example Request
POST https://ads-api.x.com/12/accounts/18ce54d4x5t/ab_tests -d '{"end_time": "2022-05-30T01:00:00Z", "entity_type" : "CAMPAIGN", "start_time": "2022-05-25T01:00:00Z", "user_groups": [{"entity_ids": ["f2qcw", "f2tht"], "size": "50.00", "name": "first group"},{"entity_ids": ["f2rqi", "f2tws"], "size": "50.00", "name": "second group", "description": "second AB test group"}], "name": "first AB test", "description": "documentation example"}'
Example Response
PUT accounts/:account_id/ab_tests/:ab_test_id
Update the specified A/B Test.
All parameters are sent in the request body and a Content-Type
of application/json
is required.
This endpoint supports partial JSON with object IDs. The following principles apply:
- To add or remove objects or elements, pass in the entire array (and its substructures); this is a replacement operation
- Think of this as recreating the array
- Otherwise, modify (change, add, remove) existing fields by referencing key names or IDs
- To remove a field, set its value to
null
- Fields that are not passed in are not modified
- To remove a field, set its value to
In general, A/B Tests can only be updated while the status
is SCHEDULED
. There is one exception: it’s possible to update the A/B Test’s end_time
while it’s LIVE
.
Resource URL
https://ads-api.x.com/12/accounts/18ce54d4x5t/:ab_test_id
Parameters
Name | Description |
---|---|
account_id required | The identifier for the leveraged account. Appears within the resource’s path and is generally a required parameter for all Advertiser API requests excluding GET accounts. The specified account must be associated with the authenticated user. Type: string Example: 18ce54d4x5t |
ab_test_id required | A reference to the A/B Test you are operating with in the request. Type: string Example: hr7l0 |
description optional | The description for the A/B Test. Maximum length: 1,024 characters. Note: Can only be updated while the A/B Test status is SCHEDULED .Type: string Example: documentation example |
end_time optional | The time, expressed in ISO 8601, that the A/B Test will end. Note: Can only be updated while the A/B Test status is SCHEDULED or LIVE .Type: string Example: 2020-10-02T00:00:00Z |
name optional | The name for the A/B Test. Maximum length: 255 characters. Note: Can only be updated while the A/B Test status is SCHEDULED .Type: string Example: first AB test |
start_time optional | The time, expressed in ISO 8601, that the A/B Test will begin. Note: Can only be updated while the A/B Test status is SCHEDULED .Type: string Example: 2022-05-30T00:00:00Z |
user_groups required | Describes the user groups. More information in the table below. Note: Can only be updated while the A/B Test status is SCHEDULED .Type: array of objects |
User Groups
Name | Description |
---|---|
id sometimes required | A reference to the user group object you are operating with in the request. Note: Required when modifying (changing, adding, or removing) user group object fields. Note: Do not specify an ID when adding or removing entire user group objects. Type: string Example: p1bcx |
description optional | The description for the user group. Maximum length: 1,024 characters. Note: Unset (remove) by specifying the field with a null value.Type: string Example: second AB test group |
entity_ids optional | An array of entity IDs. Note: This is a replacement operation. It overwrites whatever was previously set. Note: Entities can only be associated with one A/B Test. Type: array Example: ["dxi0l", "e66bl"] |
name optional | The name for the user group. Maximum length: 255 characters. Note: Unset (remove) by specifying the field with a null value.Type: string Example: first group |
size optional | The percentage of users to allocate to this user group. This is a numeric value represented as a string with at most two digits after the decimal point. For example, represent 40% as either: 40, 40.0, or 40.00. Note: The size values across objects must add up to 100.00. Type: string Min, Max: 1.00 , 99.00 |
Example Request
This request makes the following modifications.
- Removes the A/B Test description
- Changes the end time
- Adds a description to the first user group
- Changes the percentage of users in each user group
- Add an entity ID (
f2syz
) to the second user group
PUT https://ads-api.x.com/12/accounts/18ce54d4x5t/ab_tests/hr7l0 -d '{"description": null, "end_time": "2022-06-01T01:00:00Z", "user_groups": [{"id": "p1bcx", "description": "first AB test group", "size": "60.00"},{"id": "p1bcy", "size": "40.00", "entity_ids": ["f2rqi", "f2tws", "f2syz"]}]}'
Example Response
DELETE accounts/:account_id/ab_tests/:ab_test_id
Delete the specified A/B Test.
Note: Deleting an A/B Test is not reversible and subsequent attempts to delete the resource will return HTTP 404.
Resource URL
https://ads-api.x.com/12/accounts/:account_id/ab_tests/:ab_test_id
Parameters
Name | Description |
---|---|
ab_test_id required | A reference to the A/B Test you are operating with in the request. Type: string Example: hr7l0 |
Example Request
DELETE https://ads-api.x.com/12/accounts/18ce54d4x5t/ab_tests/hr7l0