Data enrichments: Enterprise

​

Introduction

​

Expanded and Enhanced URLs

• To resolve a shortened link, our system sends HTTP HEAD requests to the URL provided and follows any redirects until it arrives at the final URL. This final URL (NOT the content of the page itself) is then included in the response payload.
• The URL enrichment does add between 5-10 seconds latency to realtime streams
• For requests made to the Full Archive Search API, expanded URL enrichment data is only available for Posts 13 months old or newer.
• The URL enrichment is not available for Post links (including quote Tweets), Moments links, and profile links that are included within a Post.   

​

Post payload

• Expanded URL - unwound.url
• Expanded HTTP Status - unwound.status
• Expanded URL HTML title - 300 character limit - unwound.title
• Expanded URL HTML description - 1000 character limit - unwound.description

"entities": {
    "hashtags": [

    ],
    "urls": [
      {
        "url": "https:\/\/t.co\/HkTkwFq8UT",
        "expanded_url": "http:\/\/bit.ly\/2wYTb9y",
        "display_url": "bit.ly\/2wYTb9y",
        "unwound": {
          "url": "https:\/\/www.forbes.com\/sites\/laurencebradford\/2016\/12\/08\/11-websites-to-learn-to-code-for-free-in-2017\/",
          "status": 200,
          "title": "11 Websites To Learn To Code For Free In 2017",
          "description": "It\u2019s totally possible to learn to code for free...but what are the best resources to achieve that? Here are 11 websites where you can get started."
        },
        "indices": [
          10,
          33
        ]
      }
    ],
    "user_mentions": [

    ],
    "symbols": [

    ]
  },

"entities": {
  "urls": [{
    "url": "https://t.co/SywNbZdDmb",
    "expanded_url": "https://x.com/XDevelopers/status/1050118621198921728",
    "display_url": "x.com/XDevelopers/s…",
     "indices": [
        142,
        165
     ]
   }
  ]
}

​

Filtering operators

• Example: “url:tennis”
• Tokenized match on any Expanded URL that includes the word tennis
• Could also be used as a filter to include or exclude links from a specific website using something like “url:npr.org”

• Example: “url_title:tennis”
• Tokenized match on any Expanded URL HTML title that includes the word tennis
• Matches on the HTML title data included in the payload, which is limited to 300 characters.

• Example: “url_description:tennis”
• Tokenized match on any Expanded URL HTML description that includes the word tennis
• Matches on the HTML description included in the payload, which is limited to 1000 characters.  

​

HTTP Status Codes

• Hit a 200 series code (success)
• Hit a non-redirect series code (failures)
• Timeout because the final URL could not be resolved in a reasonable amount of time (returns a 408 - timeout)
• Hit an exception of some sort  

​

Matching rules

"matching_rules": [{
        "tag": null,
        "id": 907728589029646336,
        "id_str": "907728589029646336"
    }]

​

Poll metadata

Important Details:

• Available across all enterprise APIs (PowerTrack, Replay, Search, Historical PowerTrack)
  • Note: For Replay and Historical PowerTrack, this metadata was first made available on 02/22/17.
• Does not include vote information or poll results
• Does not currently have filter/operator support
• Available in enriched native format only
  • Enriched native format is a user-controlled setting that can be changed at any time through the Console: Select a Product (PowerTrack, Replay, Search) > Settings tab > Output Format (Leave data in its original format)

​

Post Payload

• An “options” array with up to four options that include the position (1-4) and option text
• Poll expiration date
• Poll duration

​

Sample Payload

"entities":{
          "hashtags":[],
          "urls":[],
          "user_mentions":[],
          "symbols":[],
          "polls":[
             {
                "options":[
                   {
                      "position":1,
                      "text":"The better answer"
                   },
                   {
                      "position":2,
                      "text":"The best answer"
                   }
                ],
                "end_datetime":"Sat Feb 04 15:33:11 +0000 2017",
                "duration_minutes":1440
             }
          ]
       },

​

Profile Geo

​

Introduction

​

Profile Geo Data

​

Sample Payload

{
    "user": {
        "derived": {
            "locations": \[
                {
                    "country": "United States",
                    "country_code": "US",
                    "locality": "Birmingham",
                    "region": "Alabama",
                    "sub_region": "Jefferson County",
                    "full_name": "Birmingham, Alabama, United States",
                    "geo": {
                        "coordinates": \[
                            -86.80249,
                            33.52066
                        \],
                        "type": "point"
                    }
                }
            \]
        }
    }
}

​

Limitations

• The Profile Geo enrichment attempts to determine the best choice for the geographic place described in the profile location string. The result may not be accurate in all cases due to factors such as multiple places with similar names or ambiguous names.
• If a value is not provided in a user’s profile location field (actor.location), we will not attempt to make a classification.
• Precision Level: If a Profile Geo Enrichment can only be determined with confidence at the country or region level, lower-level geographies such as subRegion and locality will be omitted from the payload.
• The Profile Geo enrichment provides lat/long coordinates (a single point) that corresponds to the Precision Level of the enrichment’s results. These coordinates represent the geographic center of the enrichment location results. For example, if the Precision Level is at the Country level, then those coordinates are set to the geographic center of that Country.
• The PowerTrack operators provided for address properties (locality/region/country/country code) are intentionally granular to allow for many rule combinations. When attempting to target a specific location that shares a name with another location, consider combining address rules. For instance, the following would avoid matches for “San Francisco, Philippines”: profile_locality:”San Francisco” profile_region:California When building rules that target individual Profile Geo fields, keep in mind that each increased level of granularity will limit the results you see. In some cases when attempting to look at data from a city, you may wish to only rely on a region rule where the region offers significant overlap with the city, e.g. the city of Zurich, Switzerland can be effectively targeted along with surrounding areas with profile_region:”Zurich”.
• Use with Native Geo Posts: The Profile Geo enrichment provides an alternative type of geography for a Post, different from the native geo value in the payload. These two types of geography can be combined together to capture all of the possible posts related to a given area (based on available geodata), though they are not conceptually equivalent.