Data Formation Migration

​

Introduction

With the launch of the v2 version of the X API, we have adopted a new data response format and method of requesting different objects and fields, which we are simply calling the X API v2 format. 

In the general differences section, you can learn about some changes that are relevant to standard, and enterprise users. However, we also put together a specific guide for the standard v1.1 Native format, the enterprise Native Enriched format, and the enterprise Activity Streams format which helps to map fields and explains which fields and expansions you must use to request the new v2 fields. 

• Native format to X API v2 (standard v1.1) 
• Native Enriched to X API v2 (enterprise)
• Activity Streams to X API v2 (enterprise)

You may also be interested in our visual data format migration tool to help you quickly see the differences between the X API v1.1 data format and the X API v2 format.

​

General differences

​

Requesting objects and fields

One of the biggest changes between the pre-v2 endpoints and v2 is that the newer version only returns a few fields by default, whereas standard, premium, and enterprise endpoints deliver most fields by default. The new version uses parameters called fields and expansions to specifically request additional data beyond the defaults, meaning that you can request just the data you need without having to ingest fields that don’t matter to you. 

Any fields that you request that relate to the primary data object will return in that primary data object along with the default values. However, if you request any expanded objects using the expansions parameter, the secondary objects will return in a new includes object. You can match the expanded objects in the includes object back to the primary object by using the ID field which will return in both.

For example, if you are using the v2 Post lookup endpoint and you include the expansions=author_id parameter in your request, you will receive the author_id field within the primary Post object, as well as a user object per Post in the includes object, each of which will include the default id field that can be used to match the user object back to the Post object. Here is an example of what this looks like:

{
  "data": [
    {
      "author_id": "2244994945",
      "id": "1397568983931392004",
      "text": "The Twitter Developer Platform. Ooh la la! https://t.co/iGTdPXBfOv https://t.co/Ze8z8EODdg"
    }
  ],
  "includes": {
    "users": [
      {
        "id": "2244994945",
        "name": "Twitter Dev",
        "username": "TwitterDev"
      }
    ]
  }
}

​

Updated JSON design

In addition to the changes in how you request certain fields, X API v2 is also introducing new JSON designs for the objects returned by the APIs, including Post and user objects.

• At the JSON root level, the standard endpoints return Post objects in a statuses array, while X API v2 returns a data array. 
• Instead of referring to Retweeted and Quoted “statuses”, X API v2 JSON refers to Retweeted and Quoted Tweets. Many legacy and deprecated fields, such as contributors and user.translator_type are being removed. 
• Instead of using both favorites (in Post object) and favourites (in user object), X API v2 uses the term like. 
• X is adopting the convention that JSON values with no value (for example, null) are not written to the payload. Post and user attributes are only included if they have a non-null values.   

​

New v2 fields

We also introduced a new set of fields to the Post object including the following:

• A conversation_id field
• Two new annotations fields, including context and entities
• Several new metrics fields 
• A new reply_setting field, which shows you who can reply to a given Post

​

Migrating from standard v1.1’s data format to v2

If you haven’t already, we recommend that you read through the data formats migration introduction to start. You may also be interested in our visual data format migration tool to help you quickly see the differences between the X API v1.1 data format and the X API v2 format.

The standard v1.1 data format, also known as the native format, is the primary format that delivers with the standard v1.1 endpoints.

If you are using the premium product, please refer to the native enriched guide. Enterprise clients might be using native enriched or activity streams, depending on how you are set up within the Gnip console. 

​

Standard v1.1 vs v2 payload structure

The following table displays the high-level objects and format that you can expect to receive from v2 compared to the v1.1 format.

Field mapping

The following section describes which v1.1 fields map to v2 fields, as well as which v2 parameters are required to receive the new field.  

​

Tweet object

Example

​

User object

Example

​

Entities and expanded entities objects

Example

​

Place object

Example

Next step

• Learn more about fields
• Learn more about expansions
• Learn how to use fields with expansions

​

Migrating from Native Enriched data format to v2

The Native Enriched data format is used by our enterprise products.

The Native Enriched data format has been updated to provide edited Tweet metadata. To learn more about Edit Tweet metadata, check out the Edit Tweets fundamentals page.

If you are using the standard v1.1 endpoints, please refer to the standard v1.1 to v2 guide. If you are using the enterprise products with Activity Streams, we have an Activity Streams to v2 guide for you as well.

X API v2 introduces new JSON designs for Tweet and user objects.

• At the JSON root level, the Native Enriched format returns Tweet objects in a results array, while X API v2 returns a data array. 
• Instead of using both favorites (in Tweet object) and favourites (in user object), X API v2 uses the term like. 
• X is adopting the convention that JSON values with no value (for example, null) are not written to the payload. Tweet and user attributes are only included if they have non-null values. 
• All id fields in v2 will be in string format  

In addition to the changes made to the new JSON format, we also introduced a new set of fields to the Tweet object including the following:

• conversation_id
• reply_settings
• alt_text on media
• Two new annotations fields, including context and entities
• Several new metrics fields
• Several new polls fields  

Many legacy and deprecated fields are being removed:

• contributors
• Certain entities.media and extended_entities.media fields
• filter_level
• timestamp_ms
• truncated

​

Native Enriched vs v2 payload structure

The following table displays the high-level objects and format that you can expect to receive from v2 compared to the Native Enriched format.

Field mapping

The following section describes which native enriched fields map to v2 fields, as well as which v2 parameters are required to receive the new field.  

​

Tweet object

​

User object

​

Entities and expanded entities objects

​

Place object

​

Poll object

​

Migrating from Activity Streams data format to v2

The Activity Streams data format is available with our enterprise products.

The Activity Streams data format has been updated to provide edited Tweet metadata. To learn more about Edit Tweet metadata, check out the Edit Tweets fundamentals page.

If you are using the standard v1.1 endpoints, please refer to the standard v1.1 to v2 guide. If you are using the premium endpoints, or the Native Enriched format for enterprise, please refer to the Native Enriched to v2 guide.

X API v2 introduces new JSON designs for Post and user objects.

• At the JSON root level, the Activity Streams format returns Tweet objects in a results array, while X API v2 returns a data array. 
• Instead of referring to Retweeted and Quoted “activities”, X API v2 JSON refers to Retweeted and Quoted Tweets. 
• Instead of using both favorites (in Tweet object) and favourites (in user object), X API v2 uses the term like. 
• Twitter is adopting the convention that JSON values with no value (for example, null) are not written to the payload. Tweet and user attributes are only included if they have non-null values. 
• All id fields in v2 will be in string format.  

In addition to the changes made to the new JSON format, we also introduced a new set of fields to the Tweet object including the following:

• conversation_id
• reply_settings
• alt_text on media
• Two new annotations fields, including context and entities
• Several new metrics fields
• Several new polls fields  

Many legacy and deprecated fields are being removed or replaced:

• display_text_range
• generator
• gnip
• link
• objectType
• provider
• twitter_entities.symbols replaced with data.entities.cashtags
• Certain twitter_extended_entities.media and twitter_entities.media fields
• twitter_filter_level
• twitterTimeZone
• verb

​

Tweet object

​

User object

​

Poll object

​

Place object

​

Media object

​

Matching rules object

​

Visual data format migration tool

The visual data format migration tool is a web application that displays the fields that map from the X API v1.1. data format to the X API v2 format for a given Tweet or user object. Either a Tweet ID or user ID can be provided to the application to see this mapping.

Please note that you will need to log in with your Twitter account in order to use the app.