Enterprise
Enterprise enrichments are additive metadata included in the response payload of some of the data APIs. They are available in paid subscription plans only.
The table below offers a brief description of each enrichment:
Enrichment: | Description: |
---|---|
Expanded and Enhanced URLs | Automatically expands shortened URLs (e.g., bitly) that are included in the body of a Post and provides HTML Title and Description metadata from the destination page. |
Matching rules object | Indicates which rule or rules matched the Posts received. The object returns rule tag and rule ID in the response object. |
Poll metadata | Notes the presence of the poll in a Post, includes the list of poll choices, and includes both the poll duration and expiration time. |
Profile geo | Derived user profile location data including the [longitude, latitude] coordinates (where possible) and related place metadata. |
The Expanded and Enhanced URL enrichment automatically expands shortened URLs that are included in the body of a Post, and includes the resulting URL as metadata within the payload. In addition, this enrichment also provides HTML page metadata from the title
and description
of the destination page.
Important Details:
To resolve a shortened link, our system sends HTTP HEAD requests to the URL provided and follows any redirects until it arrives at the final URL. This final URL (NOT the content of the page itself) is then included in the response payload.
The URL enrichment does add between 5-10 seconds latency to realtime streams
For requests made to the Full Archive Search API, expanded URL enrichment data is only available for Posts 13 months old or newer.
The URL enrichment is not available for Post links (including quote Tweets), Moments links, and profile links that are included within a Post.
The Expanded and Enhanced URL enrichment can be found within the entities
object of the Post payload - specifically in the entitites.urls.unwound
object. It provides the following fields of metadata:
unwound.url
unwound.status
unwound.title
unwound.description
This is an example of an entities object with the URL enrichment:
**This is an example of an entities object containing a Post link that is not enriched: **
The following operators will filter and provide a tokenized match on the related fields of URL metadata:
url:
url_title:
url_description:
The expanded URL enrichment also provides the HTTP status code for the final URL we are attempting to unwind. In normal cases, this will be a 200 value. Other 400-series values indicate problems with resolving the URL.
Various status codes may be returned when attempting to unwind a URL. During the process of unwinding a URL, if we get a redirect, we will follow them indefinitely until we either:
If an exception is hit, we use the following mapping between reasons and status codes returned:
Reason | Status Code Returned |
---|---|
SSL Exceptions | 403 (Forbidden) |
Unwinding not allowed by URL | 405 |
Socket Timeout | 408 (Timeout) |
Unknown Host Exception | 404 (Not Found) |
Unsupported Operation | 404 (Not Found) |
Connect Exception | 404 (Not Found) |
Illegal Argument | 400 (Bad Request) |
Everything else | 400 (Bad Request) |
The matching rules enrichment is an object of metadata that indicates which rule or rules matched the Posts received. This is most commonly used with the PowerTrack stream.
Matching will be done via exact match on the terms contained in a rule, scanning the content of the activity with and without punctuation. Matching is not case sensitive. When the content is found to contain all terms defined in the rule, there will be a root-level a matching_rules object indicating the rule(s) that led to the match.
PowerTrack
Posts delivered through PowerTrack (realtime, Replay, and Historical) will contain the matching_rules object as follows:
In PowerTrack, the matching_rules
object reflects all rules that matched the given result. In other words, if more than one rule matches a specific Post, it will only be delivered once, but the matching_rules
element will contain all the rules that matched.
Poll metadata is a free enrichment available across all products (Realtime & Historical APIs) in enriched native format payloads. The metadata notes the presence of the poll in a Post, includes the list of poll choices, and includes both the poll duration and expiration time. This enrichment makes it easy to acknowledge the presence of a poll and enables proper rendering of a poll Post for display.
Poll Post will contain the following metadata in the “entities.polls” object in the payload:
See the sample payload below for further reference.
Below is a snippet of the enriched native format payload highlighting the added poll metadata:
Many X user profiles include public location information provided by the user. This data is returned as a normal string in user.location (see User object data dictionary). The Profile Geo Enrichment adds structured geodata relevant to the user.location value by geocoding and normalizing location strings where possible. Both latitude/longitude coordinates and related place metadata are added to user.derived.locations only when included as part of the Post payload in enterprise API products. This data is available when using the enriched native format and can be filtered with a combination of PowerTrack rules.
Note: Some of the supporting geodata used to create the Profile Geo enrichment comes from GeoNames.org and is used by X under the Creative Commons Attribution 3.0 License.
Profile Geo data will be included in X’s PowerTrack, Replay, Volume Stream, Search, and Historical PowerTrack APIs.
Enriched native field name | Example value | Description |
---|---|---|
user.derived.locations.country | United States | The country location for where the user that created the Post is from. |
user.derived.locations.country_code | US | A two-letter ISO-3166 country code that corresponds to the country location for where the user that created the Post is from. |
user.derived.locations.locality | Birmingham | The locality location (generally city) for where the user that created the Post is from. |
user.derived.locations.region | Alabama | The region location (generally state/province) for where the user that created the Post is from. |
user.derived.locations.sub_region | Jefferson County | The sub-region location (generally county) for where the user that created the Post is from. |
user.derived.locations.full_name | Birmingham, Alabama, United States | The full name (excluding sub-region) for where the user that created the Post is from. |
User.derived.locations.geo | See Below | An array that includes a lat/long value for a coordinate that corresponds to the lowers granularity location for where the user that created the Post is from. |
The Historical PowerTrack, Search, and PowerTrack APIs supports filtering based on Profile Geo data. See the appropriate product documentation for more details on what operators are available for filtering on Profile Geo data.
Enterprise
Enterprise enrichments are additive metadata included in the response payload of some of the data APIs. They are available in paid subscription plans only.
The table below offers a brief description of each enrichment:
Enrichment: | Description: |
---|---|
Expanded and Enhanced URLs | Automatically expands shortened URLs (e.g., bitly) that are included in the body of a Post and provides HTML Title and Description metadata from the destination page. |
Matching rules object | Indicates which rule or rules matched the Posts received. The object returns rule tag and rule ID in the response object. |
Poll metadata | Notes the presence of the poll in a Post, includes the list of poll choices, and includes both the poll duration and expiration time. |
Profile geo | Derived user profile location data including the [longitude, latitude] coordinates (where possible) and related place metadata. |
The Expanded and Enhanced URL enrichment automatically expands shortened URLs that are included in the body of a Post, and includes the resulting URL as metadata within the payload. In addition, this enrichment also provides HTML page metadata from the title
and description
of the destination page.
Important Details:
To resolve a shortened link, our system sends HTTP HEAD requests to the URL provided and follows any redirects until it arrives at the final URL. This final URL (NOT the content of the page itself) is then included in the response payload.
The URL enrichment does add between 5-10 seconds latency to realtime streams
For requests made to the Full Archive Search API, expanded URL enrichment data is only available for Posts 13 months old or newer.
The URL enrichment is not available for Post links (including quote Tweets), Moments links, and profile links that are included within a Post.
The Expanded and Enhanced URL enrichment can be found within the entities
object of the Post payload - specifically in the entitites.urls.unwound
object. It provides the following fields of metadata:
unwound.url
unwound.status
unwound.title
unwound.description
This is an example of an entities object with the URL enrichment:
**This is an example of an entities object containing a Post link that is not enriched: **
The following operators will filter and provide a tokenized match on the related fields of URL metadata:
url:
url_title:
url_description:
The expanded URL enrichment also provides the HTTP status code for the final URL we are attempting to unwind. In normal cases, this will be a 200 value. Other 400-series values indicate problems with resolving the URL.
Various status codes may be returned when attempting to unwind a URL. During the process of unwinding a URL, if we get a redirect, we will follow them indefinitely until we either:
If an exception is hit, we use the following mapping between reasons and status codes returned:
Reason | Status Code Returned |
---|---|
SSL Exceptions | 403 (Forbidden) |
Unwinding not allowed by URL | 405 |
Socket Timeout | 408 (Timeout) |
Unknown Host Exception | 404 (Not Found) |
Unsupported Operation | 404 (Not Found) |
Connect Exception | 404 (Not Found) |
Illegal Argument | 400 (Bad Request) |
Everything else | 400 (Bad Request) |
The matching rules enrichment is an object of metadata that indicates which rule or rules matched the Posts received. This is most commonly used with the PowerTrack stream.
Matching will be done via exact match on the terms contained in a rule, scanning the content of the activity with and without punctuation. Matching is not case sensitive. When the content is found to contain all terms defined in the rule, there will be a root-level a matching_rules object indicating the rule(s) that led to the match.
PowerTrack
Posts delivered through PowerTrack (realtime, Replay, and Historical) will contain the matching_rules object as follows:
In PowerTrack, the matching_rules
object reflects all rules that matched the given result. In other words, if more than one rule matches a specific Post, it will only be delivered once, but the matching_rules
element will contain all the rules that matched.
Poll metadata is a free enrichment available across all products (Realtime & Historical APIs) in enriched native format payloads. The metadata notes the presence of the poll in a Post, includes the list of poll choices, and includes both the poll duration and expiration time. This enrichment makes it easy to acknowledge the presence of a poll and enables proper rendering of a poll Post for display.
Poll Post will contain the following metadata in the “entities.polls” object in the payload:
See the sample payload below for further reference.
Below is a snippet of the enriched native format payload highlighting the added poll metadata:
Many X user profiles include public location information provided by the user. This data is returned as a normal string in user.location (see User object data dictionary). The Profile Geo Enrichment adds structured geodata relevant to the user.location value by geocoding and normalizing location strings where possible. Both latitude/longitude coordinates and related place metadata are added to user.derived.locations only when included as part of the Post payload in enterprise API products. This data is available when using the enriched native format and can be filtered with a combination of PowerTrack rules.
Note: Some of the supporting geodata used to create the Profile Geo enrichment comes from GeoNames.org and is used by X under the Creative Commons Attribution 3.0 License.
Profile Geo data will be included in X’s PowerTrack, Replay, Volume Stream, Search, and Historical PowerTrack APIs.
Enriched native field name | Example value | Description |
---|---|---|
user.derived.locations.country | United States | The country location for where the user that created the Post is from. |
user.derived.locations.country_code | US | A two-letter ISO-3166 country code that corresponds to the country location for where the user that created the Post is from. |
user.derived.locations.locality | Birmingham | The locality location (generally city) for where the user that created the Post is from. |
user.derived.locations.region | Alabama | The region location (generally state/province) for where the user that created the Post is from. |
user.derived.locations.sub_region | Jefferson County | The sub-region location (generally county) for where the user that created the Post is from. |
user.derived.locations.full_name | Birmingham, Alabama, United States | The full name (excluding sub-region) for where the user that created the Post is from. |
User.derived.locations.geo | See Below | An array that includes a lat/long value for a coordinate that corresponds to the lowers granularity location for where the user that created the Post is from. |
The Historical PowerTrack, Search, and PowerTrack APIs supports filtering based on Profile Geo data. See the appropriate product documentation for more details on what operators are available for filtering on Profile Geo data.