​

Introduction

The filtered stream endpoint group enables developers to filter the real-time stream of public Posts. This endpoint group’s functionality includes multiple endpoints that enable you to create and manage rules, and apply those rules to filter a stream of real-time Posts that will return matching public Posts. This endpoint group allows users to listen for specific topics and events in real-time, monitor the conversation around competitions, understand how trends develop in real-time, and much more.

Developers can use the REST rules endpoint to add and remove rules to a persistent stream connection without needing to disconnect. These rules can be created with operators that match on Post attributes such as message keywords, hashtags, and URLs. Operators and rule clauses can be combined with boolean logic and parentheses to help refine the filter’s matching behavior. 

Once you’ve added a set of rules, you can establish a streaming connection which will start to deliver Post objects in JSON format through a persistent HTTP Streaming connection. You will only receive content matching your rules while connected to the stream.

The filtered search endpoint supports edited Posts. This endpoint will deliver edited Posts that match one or more of your filters, along with its edit history, including an array of Post IDs. For Posts with no edit history, this array will hold a single ID. For Posts that have been edited, this array contains multiple IDs, arranged in ascending order reflecting the order of edits, with the most recent version in the last position of the array. To learn more about how Post edits work, see the Posts edits fundamentals page. 

Certain aspects of the filtered stream endpoint are limited by access level:

Pro access

• 1000 rules per stream
• 100 requests per 15 minutes when using the POST /2/tweets/search/stream/rules endpoint to add rules
• Can use all operators when building your rule
• Can build rules up to 1024 characters in length

Enterprise access

• 25,000+ rules per stream
• Can use all operators when building your rule
• Can build rules up to 2048 characters in length
• Apply here for Enterprise access

The returned Posts from filtered stream count towards the monthly Post cap.

​

Getting started with the filtered stream endpoints

This quick start guide will help you make your first request to the filtered stream endpoint group using a cURL request. cURL is a command line tool which allows you to make requests with minimal configuration.

If you would like to see sample code in different languages, please visit ourX API v2 sample code GitHub repository. 

​

Steps to build a filtered stream request using cURL

Step one: Create a rule

Rules are made up of one or many different operators that are combined using boolean logic and parentheses to help define which Posts will deliver to your stream. In this guide, we will filter the stream to find Posts that contain both the keyword “cat” and images. Here is our rule:

cat has:images

Step two: Add a tag to your rule

You can add multiple concurrent rules to your stream. When you open your streaming connection, Posts that match any of these rules will flow through the same streaming connection. To ensure that you know which Post matches which rule, you can pass a tag along with your rule creation request. Each Post that matches that rule will then include a tag field within the Post payload noting which rule it matched. For this rule, we are going to assign the following tag:

cats with images

Step three: Add your rule to the stream

This endpoint requires you to pass an application/JSON body along with this request that contains your rule and its tag. You will also notice that we have included the rule value and tag within an add object since we are trying to add this rule to the stream. This JSON body will look like this:

{
  "add": \[
    {"value": "cat has:images", "tag": "cats with images"}
  \]
}

Now that you have fully set up this JSON body, you can add that to a cURL request, which will look like the following. This request isn’t ready yet, so please hold off on submitting it until a later step.

curl -X POST 'https://api.x.com/2/tweets/search/stream/rules' \
-H "Content-type: application/json" \
-H "Authorization: Bearer $APP_ACCESS_TOKEN" -d \

'{
  "add": [
    {"value": "cat has:images", "tag": "cats with images"}
  ]
}'`

Step four: Authenticate your request

Since the filtered stream rules endpoints require OAuth 2.0 App-Only authentication, you will need to replace $APP_ACCESS_TOKEN in the cURL command from step three with the App Access Token that you generated in the prerequisites.   

Step five: Add your rule to the stream

Next step is to execute your cURL request, which will add the rule to your stream. Copy and paste the cURL command into your command line interface and press “return”. 

If your cURL request was successful, you’ll receive a response containing data about your value, tag, and id (serves as a unique identifier for your rule). This response will look similar to this:  

{
	"data": [{
		"value": "cat has:images",
		"tag": "cats with images",
		"id": "1273026480692322304"
	}],
	"meta": {
		"sent": "2020-06-16T22:55:39.356Z",
		"summary": {
			"created": 1,
			"not_created": 0,
			"valid": 1,
			"invalid": 0
		}
	}
}

You can validate that your rule was added successfully by sending the following GET request to the rules endpoint, once again making sure to replace $APP_ACCESS_TOKEN with your token. This request will return a full list of all the rules that have been added to your stream.

-H "Authorization: Bearer $APP_ACCESS_TOKEN"

If your cURL request was successful, you should receive the following:  

	"data": [{
		"id": "1273028376882589696",
		"value": "cat has:images",
		"tag": "cats with images"
	}],
	"meta": {
		"sent": "2020-06-16T23:14:06.498Z"
	}
}

​

Step six: Identify and specify which fields you would like to retrieve

If you connect to the stream after step five, you will receive the default Post object fields in your response: id , text, and edit_history_tweet_ids. If you would like to receive additional fields beyond these, you will have to specify those fields in your request with the field and/or expansion parameters.

For this exercise, we will request a three different sets of fields from different objects:

1. The additional tweet.created_at field in the primary Post objects.

2. The associated authors’ user object’s default fields for the returned Posts: id, name, and username

3. The additional  user.created_at field in the associated user objects.  

To request these fields, you will need to pass the following in your request:

Now that you know this, you can piece together your request URL to connect to the stream, which will look like this:  

https://api.x.com/2/tweets/search/stream?tweet.fields=created_at&expansions=author_id&user.fields=created_at

​

Step seven: Connect to the stream and review your response

Now that your rule is in place and you’ve specified which fields you want returned, you’re ready to connect to the stream, which will deliver Post objects that match the rule you’ve submitted. This is what the cURL command looks like once you’ve added the URL from step six into your request:  

curl -X GET -H "Authorization: Bearer $APP_ACCESS_TOKEN" "https://api.x.com/2/tweets/search/stream?tweet.fields=created_at&expansions=author_id&user.fields=created_at"

Once again, this request must be authenticated using OAuth 2.0 App-Only, so make sure to replace $APP_ACCESS_TOKEN with your credentials before copying and pasting it into your command line tool.

Once you are connected to the filtered stream you will start to receive Posts that match your rules in the following JSON format:  

{
  "data": [
    {
      "author_id": "2244994945",
      "created_at": "2022-09-14T19:00:55.000Z",
      "id": "1228393702244134912",
      "edit_history_tweet_ids": ["1228393702244134912"],
      "text": "What did the developer write in their Valentine’s card?\n  \nwhile(true) {\n    I = Love(You);  \n}"
    },
    {
      "author_id": "2244994945",
      "created_at": "2022-09-12T17:09:56.000Z",
      "id": "1227640996038684673",
       "edit_history_tweet_ids": ["1227640996038684673"],
      "text": "Doctors: Googling stuff online does not make you a doctor\n\nDevelopers: https://t.co/mrju5ypPkb"
    },
    {
      "author_id": "2244994945",
      "created_at": "2022-09-27T20:26:41.000Z",
      "id": "1199786642791452673",
      "edit_history_tweet_ids": ["1199786642791452673"],
      "text": "C#"
    }
  ],
  "includes": {
    "users": [
      {
        "created_at": "2013-12-14T04:35:55.000Z",
        "id": "2244994945",
        "name": "Twitter Dev",
        "username": "TwitterDev"
      }
    ]
  }
}

If you would like to close your connection, you can press Control-C in your command line tool on either Mac or Windows systems to break the connection, or you can also close the window. 

​

Integrate

​

Building rules for filtered stream

The filtered stream endpoints deliver filtered Posts to you in real-time that match on a set of rules that are applied to the stream. Rules are made up of operators that are used to match on a variety of Post attributes.

Multiple rules can be applied to a stream using the POST /tweets/search/stream/rules endpoint. Once you’ve added rules and connect to your stream using the GET /tweets/search/stream endpoint, only those Posts that match your rules will be delivered in real-time through a persistent streaming connection. You do not need to disconnect from your stream to add or remove rules. 

To learn more about how to create high-quality rules, visit the following tutorial: Building high-quality filters for getting X data  

​

Table of contents

• Building a rule
  • Rule limitations
  • Operator availability
  • Operator types: standalone and conjunction-required
  • Boolean operators and grouping
  • Order of operations
  • Punctuation, diacritics, and case sensitivity
  • Specificity and efficiency
  • Iteratively building a rule
  • Adding and removing rules
  • Rule examples
• List of operators  

​

Building a rule

Rule limitations

Limits on the number of rules will depend on which access level is applied to your Project.

You can see how these limits apply via the filtered stream introduction page.  

Operator availability

While most operators are available to any developer, there are several that are reserved for those that have been approved for Basic, Pro, or Enterprise access levels. We list which access level each operator is available to in the list of operators table using the following labels:

• Essential operators: Available when using any access level.
• Elevated operators: Available when using a Project with Pro, or Enterprise access.  

Operator types: standalone and conjunction-required

Standalone operators can be used alone or together with any other operators (including those that require conjunction).

For example, the following rule will work because it uses the #hashtag operator, which is standalone:

#xapiv2

Conjunction required operators cannot be used by themselves in a rule; they can only be used when at least one standalone operator is included in the rule. This is because using these operators alone would be far too general, and would match on an extremely high volume of Posts.

For example, the following rules are not supported since they contain only conjunction required operators:

has:media has:links OR is:retweet

If we add in a standalone operator, such as the phrase “X data”, the rule would then work properly. 

“X data” has:mentions (has:media OR has:links)

Boolean operators and grouping

If you would like to string together multiple operators in a single rule, you have the following tools at your disposal:

Order of operations

When combining AND and OR functionality, the following order of operations will dictate how your rule is evaluated.

1. Operators connected by AND logic are combined first
2. Then, operators connected with OR logic are applied

For example:

• apple OR iphone ipad would be evaluated as apple OR (iphone ipad)
• ipad iphone OR android would be evaluated as (iphone ipad) OR android

To eliminate uncertainty and ensure that your rule is evaluated as intended, group terms together with parentheses where appropriate. 

For example:

• (apple OR iphone) ipad
• iphone (ipad OR android)  

Punctuation, diacritics, and case sensitivity

If you specify a keyword or hashtag rule with character accents or diacritics, it will match Posts that contain the exact word with proper accents or diacritics, but not those that have the proper letters, but without the accent or diacritic. 

For example, rules with the keyword diacrítica or hashtag #cumpleaños will match Posts that contain diacrítica or #cumpleaños because they include the accent or diacritic. However, these rules will not match Posts that contain Diacritica or #cumpleanos without the tilde í or eñe.

Characters with accents or diacritics are treated the same as normal characters and are not treated as word boundaries. For example, a rule with the keyword cumpleaños would only match Posts containing the word cumpleaños and would not match Posts containing cumplea, cumplean, or os.

All operators are evaluated in a case-insensitive manner. For example, the rule cat will match all of the following: cat, CAT, Cat.

Specificity and efficiency

When you start to build your rule, it is important to keep a few things in mind.

• Using broad, standalone operators for your rule such as a single keyword or #hashtag is generally not recommended since it will likely match on a massive volume of Posts. Creating a more robust rule will result in a more specific set of matching Posts, and will hopefully reduce the amount of noise in the payload that you will need to sift through to find valuable insights. 
  • For example, if your rule was just the keyword happy you will likely get anywhere from 200,000 - 300,000 Posts per day.
  • Adding more conditional operators narrows your search results, for example (happy OR happiness) place_country:GB -birthday -is:retweet
• Writing efficient rules is also beneficial for staying within the characters rule length restriction. The character count includes the entire rule string including spaces and operators.
  • For example, the following rule is 59 characters long: (happy OR happiness) place_country:GB -birthday -is:retweet

Quote Tweet matching behavior

When using the filtered stream endpoints, operators will match on both the content from the original Post that was quoted, as well as the content included in the Quote Tweet.

However, please note that the Search Posts endpoints will not match on the content from the original Post that was quoted, but will match on the Quote Tweet’s content.

Iteratively building a rule

Test your rule early and often

Getting a rule to return the “right” results the first time is rare. There is so much on X that may or may not be obvious at first and the rule syntax described above may be hard to match to your desired search. As you build a rule, it is important for you to periodically test it out with the stream endpoint to see what data it returns. You can also test with one of the Search Post endpoints, assuming the operators that you are using are also available via that endpoint. 

For this section, we are going to start with the following rule and adjust it based on the results that we receive during our test: 

happy OR happiness

Use results to narrow the rule

As you test the rule, you should scan the returned Posts to see if they include the data that you are expecting and hoping to receive. Starting with a broad rule and a superset of Post matches allows you to review the result and narrow the rule to filter out undesired results.  

When we tested the example rule, we noticed that we were getting Posts in a variety of different languages. In this situation, we want to only receive Posts that are in english, so we’re going to add the lang: operator:

(happy OR happiness) lang:en

The test delivered a number of Posts wishing people a happy birthday, so we are going to add -birthday as a negated keyword operator. We also want to only receive original Posts, so we’ve added the negated -is:retweet operator:

(happy OR happiness) lang:en -birthday -is:retweet

Adjust for inclusion where needed

If you notice that you are not receiving data that you expect and know that there are existing Posts that should return, you may need to broaden your rule by removing operators that may be filtering out the desired data. 

For our example, we noticed that there were other Posts in our personal timeline that expressed the emotion that we are looking for and weren’t included in the test results. To ensure we have greater coverage, we are going to add the keywords, excited and elated.

(happy OR happiness OR excited OR elated) lang:en -birthday -is:retweet

Adjust for popular trends/bursts over the time period

Trends come and go on X quickly. Maintaining your rule should be an active process. If you plan to use a single rule for a while, we suggest that you periodically check in on the data that you are receiving to see if you need to make any adjustments.

In our example, we notice that we started to receive some Posts that are wishing people a “happy holidays”. Since we don’t want these Posts included in our results, we are going to add a negated -holidays keyword.

(happy OR happiness OR excited OR elated) lang:en -birthday -is:retweet -holidays 

Adding and removing rules

You will be using the POST /2/tweets/search/stream/rules endpoint when both adding and deleting rules from your stream.

To add one or more rule to your stream, submit an add JSON body with an array that contains the value parameter including the rule, and the optional tag parameter including free-form text that you can use to identify which returned Posts match this rule. 

For example, if you were trying to add a set of rules to your stream, your cURL command might look like this:

`curl -X POST ‘https://api.x.com/2/tweets/search/stream/rules’
-H “Content-type: application/json”
-H “Authorization: Bearer $ACCESS_TOKEN” -d \

'{
  "add": [
    {"value": "cat has:media", "tag": "cats with media"},
    {"value": "cat has:media -grumpy", "tag": "happy cats with media"},
    {"value": "meme", "tag": "funny things"},
    {"value": "meme has:images"}
  ]
}'

Similarly, to remove one or more rules from your stream, submit a delete JSON body with the array of that contains the id parameter including the rule IDs that you would like to delete.

For example, if you were trying to remove a set of rules from your stream, your cURL command might look like this:

`curl -X POST ‘https://api.x.com/2/tweets/search/stream/rules’
-H “Content-type: application/json”
-H “Authorization: Bearer $ACCESS_TOKEN” -d \

'{
  "delete": {
    "ids": [
      "1165037377523306498",
      "1165037377523306499"
    ]
  }
}'

We have sample code in different languages available via our Github. 

​

Rule examples

Tracking a natural disaster

The following rule matched on Posts coming from weather agencies and gauges that discuss Hurricane Harvey, which hit Houston in 2017. Notice the use of the matching rules tag, and the JSON format that you will need to use when submitting the rule to the POST /2/tweets/search/stream/rules endpoint.

{
      "value": "-is:retweet has:geo (from:NWSNHC OR from:NHC_Atlantic OR from:NWSHouston OR from:NWSSanAntonio OR from:USGS_TexasRain OR from:USGS_TexasFlood OR from:JeffLindner1)",
      "tag": "theme:info has:geo original from weather agencies and gauges"
  }

Reviewing the sentiment of a conversation

The next rule could be used to better understand the sentiment of the conversation developing around the hashtag, #nowplaying, but only from Posts published within North America.

  {
      "value": "#nowplaying (happy OR exciting OR excited OR favorite OR fav OR amazing OR lovely OR incredible) (place_country:US OR place_country:MX OR place_country:CA) -horrible -worst -sucks -bad -disappointing",
      "tag": "#nowplaying positive"
  },
  {
      "value": "#nowplaying (horrible OR worst OR sucks OR bad OR disappointing) (place_country:US OR place_country:MX OR place_country:CA) -happy -exciting -excited -favorite -fav -amazing -lovely -incredible",
      "tag": "#nowplaying negative"
  }

Find Posts that relate to a specific Post annotation

This rule was built to search for original Posts that included an image of a pet that is not a cat, where the language identified in the Post is Japanese. To do this, we used the context: operator to take advantage of the Post annotation functionality. We first used the Post lookup endpoint and the tweet.fields=context_annotations fields parameter to identify which domain.entity IDs we need to use in our query:

• Posts that relate to cats return domain 66 (Interests and Hobbies category) with entity 852262932607926273 (Cats). 
• Posts that relate to pets return domain 65 (Interests and Hobbies Vertical) with entity 852262932607926273 (Pets).   

Here is what the rule would look like:

      "value": "context:65.852262932607926273 -context:66.852262932607926273 -is:retweet has:images lang:ja",
      "tag": "Japanese pets with images - no cats"
  }

​

Operators

• **Essential: **Available when using any access level.
• **Elevated: **Available when using a Project with Pro, or Enterprise access.
• For some operators, an alternate name, or alias, is available. 

OperatorTypeAvailabilityDescriptionkeywordStandalone

Essential

Matches a keyword within the body of a Post. This is a tokenized match, meaning that your keyword string will be matched against the tokenized text of the Post body. Tokenization splits words based on punctuation, symbols, and Unicode basic plane separator characters.

For example, a Post with the text “I like coca-cola” would be split into the following tokens: I, like, coca, cola. These tokens would then be compared to the keyword string used in your rule. To match strings containing punctuation (for example coca-cola), symbol, or separator characters, you must wrap your keyword in double-quotes.

Example: pepsi OR cola OR “coca cola”emojiStandaloneEssentialMatches an emoji within the body of a Post. Similar to a keyword, emojis are a tokenized match, meaning that your emoji will be matched against the tokenized text of the Post body.

Note that if an emoji has a variant, you must wrap it in double quotes to add to a rule.

Example: (😃 OR 😡) 😬“exact phrase match”StandaloneEssentialMatches the exact phrase within the body of a Post.

Example: (“X API” OR #v2) -“filtered stream”#StandaloneEssentialMatches any Post containing a recognized hashtag, if the hashtag is a recognized entity in a Post.

This operator performs an exact match, NOT a tokenized match, meaning the rule #thanku will match posts with the exact hashtag #thanku, but not those with the hashtag #thankunext.

Example: #thankunext #fanart OR @arianagrande@StandaloneEssentialMatches any Post that mentions the given username, if the username is a recognized entity (including the @ character).

Example: (@XDeveloeprs OR @api) -@x$StandaloneEssentialMatches any Post that contains the specified ‘cashtag’ (where the leading character of the token is the ‘$’ character).

Note that the cashtag operator relies on X’s ‘symbols’ entity extraction to match cashtags, rather than trying to extract the cashtag from the body itself.

Example: $twtr OR @XDevelopers -$fbfrom:StandaloneEssentialMatches any Post from a specific user. The value can be either the username (excluding the @ character) or the user’s numeric user ID.

You can only pass a single username/ID from: operator.

Example: from:XDevelopers OR from:api -from:Xto:StandaloneEssentialMatches any Post that is in reply to a particular user. The value can be either the username (excluding the @ character) or the user’s numeric user ID.

You can only pass a single username/ID per to: operator.

Example: to:XDevelopers OR to:api -to:xurl:StandaloneEssentialPerforms a tokenized match on any validly-formatted URL of a Post.

This operator can matches on the contents of both the url or expanded_url fields. For example, a Post containing “You should check out X Developer Labs: https://t.co/c0A36SWil4” (with the short URL redirecting to https://developer.twitter.com) will match both the following rules:

from:XDevelopers url:“https://developer.twitter.com” (because it will match the contents of entities.urls.expanded_url)

from:XDevelopers url:“https://t.co” (because it will match the contents of entities.urls.url)

Tokens and phrases containing punctuation or special characters should be double-quoted (for example, url:“/developer”). Similarly, to match on a specific protocol, enclose in double-quotes (for example, url:“https://developer.twitter.com”).

You can only pass a single URL per url: operator.retweets_of:StandaloneEssential

Available alias: retweets_of_user:

Matches Posts that are Retweets of the specified user. The value can be either the username (excluding the @ character) or the user’s numeric user ID.

You can only pass a single username/ID per retweets_of: operator.

Example: retweets_of:XDevelopers OR retweets_of:twitterapi

See HERE for methods for looking up numeric X Account IDs.

context:StandaloneEssentialMatches Posts with a specific domain id and/or domain id, enitity id pair where * represents a wildcard. To learn more about this operator, please visit our page on Post annotations.

You can only pass a single domain/entitie per context: operator.

context:domain_id.entity_id context:domain_id.* context:*.entity_id

Examples: context:10.799022225751871488 (domain_id.entity_id returns Posts matching that specific domain-entity pair)

context:47.* (domain_id.* returns Posts matching that domain ID, with any domain-entity pair)

context:.799022225751871488 (.entity_id returns Posts matching that entity ID, with any domain-entity pair)entity:StandaloneEssentialMatches Posts with a specific entity string value. To learn more about this operator, please visit our page on annotations.

You can only pass a single entity per entity: operator.

entity:“string declaration of entity/place”

Examples: entity:“Michael Jordan” OR entity:“Barcelona”conversation_id:StandaloneEssentialMatches Posts that share a common conversation ID. A conversation ID is set to the Post ID of a Post that started a conversation. As Replies to a Post are posted, even Replies to Replies, the conversation_id is added to its JSON payload.

You can only pass a single conversation ID per conversation_id: operator.

Example: conversation_id:1334987486343299072 (from:XDevelopers OR from:api)

bio:

StandaloneEssential

_Available alias: _user_bio:

Matches a keyword or phrase within the Post publisher’s bio. This is a tokenized match within the contents of the description field within the User object.

Example: bio:developer OR bio:“data engineer” OR bio:academic

bio_name:StandaloneEssentialMatches a keyword within the Post publisher’s user bio name. This is a tokenized match within the contents of a user’s “name” field within the User object.

Example: bio_name:phd OR bio_name:md

bio_location:

StandaloneEssential

_Available alias: _user_bio_location:

Matches Posts that are published by users whose location contains the specified keyword or phrase. This operator performs a tokenized match, similar to the normal keyword rules on the message body.

This location is part of the User object, matches on the ‘location’ field, and is a non-normalized, user-generated, free-form string. It is also different from a Post’s location (see place:).

Example: bio_location:“big apple” OR bio_location:nyc OR bio_location:manhattan

place:StandaloneElevatedMatches Posts tagged with the specified location or X place ID. Multi-word place names (“New York City”, “Palo Alto”) should be enclosed in quotes.

You can only pass a single place per place: operator.

Note: See the GET geo/search standard v1.1 endpoint for how to obtain X place IDs.

Note: This operator will not match on Retweets, since Retweet’s places are attached to the original Post. It will also not match on places attached to the original Post of a Quote Tweet.

Example: place:“new york city” OR place:seattle OR place:fd70c22040963ac7place_country:StandaloneElevatedMatches Posts where the country code associated with a tagged place/location matches the given ISO alpha-2 character code.

You can find a list of valid ISO codes on Wikipedia.

You can only pass a single ISO code per place_country: operator.

Note: This operator will not match on Retweets, since Retweet’s places are attached to the original Post. It will also not match on places attached to the original Post of a Quote Tweet.

Example: place_country:US OR place_country:MX OR place_country:CApoint_radius:StandaloneElevated

Matches against the place.geo.coordinates object of the Post when present, and in X, against a place geo polygon, where the Place polygon is fully contained within the defined region.

point_radius:[longitude latitude radius]

• Units of radius supported are miles (mi) and kilometers (km)

• Radius must be less than 25mi

• Longitude is in the range of ±180

• Latitude is in the range of ±90

• All coordinates are in decimal degrees

• Rule arguments are contained within brackets, space delimited

You can only pass a single geo polygon per point_radius: operator.

Note: This operator will not match on Retweets, since Retweet’s places are attached to the original Post. It will also not match on places attached to the original Post of a Quote Tweet.

Example: point_radius:[2.355128 48.861118 16km] OR point_radius:[-41.287336 174.761070 20mi]

bounding_box:

StandaloneElevated

_Available alias: _geo_bounding_box:

Matches against the place.geo.coordinates object of the Post when present, and in X, against a place geo polygon, where the place polygon is fully contained within the defined region.

bounding_box:[west_long south_lat east_long north_lat]

• west_long south_lat represent the southwest corner of the bounding box where west_long is the longitude of that point, and south_lat is the latitude.

• east_long north_lat represent the northeast corner of the bounding box, where east_long is the longitude of that point, and north_lat is the latitude.

• Width and height of the bounding box must be less than 25mi

• Longitude is in the range of ±180

• Latitude is in the range of ±90

• All coordinates are in decimal degrees.

• Rule arguments are contained within brackets, space delimited.

You can only pass a single geo polygons per bounding_box: operator.

Note: This operator will not match on Retweets, since Retweet’s places are attached to the original Post. It will also not match on places attached to the original Post of a Quote Tweet.

Example: bounding_box:[-105.301758 39.964069 -105.178505 40.09455]

is:retweetConjunction requiredEssentialMatches on Retweets that match the rest of the specified rule. This operator looks only for true Retweets (for example, those generated using the Retweet button). Quote Tweets will not be matched by this operator.

Example: data @XDevelopers -is:retweetis:replyConjunction requiredEssentialDeliver only explicit replies that match a rule. Can also be negated to exclude replies that match a rule from delivery.

When used with the filtered stream, this operator matches on replies to an original Post, replies in quoted Posts and replies in Retweets.

Example: from:XDevelopers is:replyis:quoteConjunction requiredEssentialReturns all Quote Tweets, also known as Posts with comments.

Example: “sentiment analysis” is:quoteis:verifiedConjunction requiredEssentialDeliver only Posts whose authors are verified by X.

Example: #nowplaying is:verified-is:nullcastConjunction requiredElevatedRemoves Posts created for promotion only on ads.twitter.com that have a “source”:“Twitter for Advertisers (legacy)” or “source”:“Twitter for Advertisers”. This operator must be negated.

For more info on Nullcasted Posts, see our page on Post availability.

Example: “mobile games” -is:nullcasthas:hashtagsConjunction requiredEssentialMatches Posts that contain at least one hashtag.

Example: from:XDevelopers -has:hashtagshas:cashtagsConjunction requiredEssentialMatches Posts that contain a cashtag symbol (with a leading ‘$’ character. For example,$tag).

Example: #stonks has:cashtagshas:linksConjunction requiredEssentialThis operator matches Posts which contain links and media in the Post body.

Example: from:XDevelopers announcement has:linkshas:mentionsConjunction requiredEssentialMatches Posts that mention another X user.

Example: #nowplaying has:mentions

has:media

Conjunction requiredEssential

_Available alias: _has:media_link

Matches Posts that contain a media object, such as a photo, GIF, or video, as determined by X. This will not match on media created with Periscope, or Posts with links to other media hosting sites.

Example: (kittens OR puppies) has:media

has:imagesConjunction requiredEssentialMatches Posts that contain a recognized URL to an image.

Example: #meme has:images

has:video_link

Conjunction requiredEssential

_Available alias: _has:videos

Matches Posts that contain native X videos, uploaded directly to X. This will not match on videos created with Periscope, or Posts with links to other video hosting sites.

Example: #icebucketchallenge has:video_link

has:geoConjunction requiredEssentialMatches Posts that have Post-specific geolocation data provided by the X user. This can be either a location in the form of a X place, with the corresponding display name, geo polygon, and other fields, or in rare cases, a geo lat-long coordinate.

Note: Operators matching on place (Post geo) will only include matches from original posts. Retweets do not contain any place data.

Example: recommend #paris has:geo -bakerysample:Conjunction requiredEssentialReturns a random percent sample of Posts that match a rule rather than the entire set of Posts. The percent value must be represented by an integer between 1 and 100 (for example, sample:10 will return a random 10% sample).

This operator first reduces the scope of the stream to the percentage you specified, then the rule/filter is applied to that sampled subset. In other words, if you are using, for example, sample:10, each Post will have a 10% chance of being in the sample.

This operator applies to the entire rule and requires all OR’d terms to be grouped.

Example: #nowplaying @spotify sample:15lang:Conjunction requiredEssentialMatches Posts that have been classified by X as being of a particular language (if, and only if, the post has been classified). It is important to note that each Post is currently only classified as being of one language, so AND’ing together multiple languages will yield no results.

You can only pass a single BCP 47 language identifier per lang: operator.

Note: if no language classification can be made the provided result is ‘und’ (for undefined).

Example: recommend #paris lang:en

The list below represents the currently supported languages and their corresponding BCP 47 language identifier:

followers_count: Essential

Matches Posts when the author has a followers count within the given range.

If a single number is specified, any number equal to or higher will match.

Example: followers_count:500

Additionally, a range can be specified to match any number in the given range.

Example: followers_count:1000..10000

tweets_count:

 Essential

_Available alias: _statuses_count:

Matches Posts when the author has posted a number of Posts that falls within the given range.

If a single number is specified, any number equal to or higher will match.

Example: tweets_count:1000

Additionally, a range can be specified to match any number in the given range.

Example: tweets_count:1000..10000

following_count:

 Essential

_Available alias: _friends_count:

Matches Posts when the author has a friends count (the number of users they follow) that falls within the given range.

If a single number is specified, any number equal to or higher will match.

Example: following_count:500

Additionally, a range can be specified to match any number in the given range.

Example: following_count:1000..10000

listed_count:

 Essential

_Available alias: _user_in_lists_count:

Matches Posts when the author is included in the specified number of Lists. 

If a single number is specified, any number equal to or higher will match.

Example: listed_count:10

Additionally, a range can be specified to match any number in the given range.

Example: listed_count:10..100

url_title:

 Essential

_Available alias: _within_url_title:

Performs a keyword/phrase match on the expanded URL HTML title metadata.

Example: url_title:snow

url_description:

 Essential

_Available alias: _within_url_description:

Performs a keyword/phrase match on the expanded page description metadata.

Example: url_description:weather

url_contains: Essential

Matches Posts with URLs that literally contain the given phrase or keyword. To search for patterns with punctuation in them (i.e. google.com) enclose the search term in quotes.

NOTE: This will match against the expanded URL as well.

Example: url_contains:photos

source: Essential

Matches any Post generated by the given source application. The value must be either the name of the application or the application’s URL. Cannot be used alone.

Example: source:“X for iPhone”

Note: As a X app developer, Posts created programmatically by your application will have the source of your application Name and Website URL set in your app settings. 

in_reply_to_tweet_id:

 Essential

_Available alias: _in_reply_to_status_id:

Deliver only explicit Replies to the specified Post.

Example: in_reply_to_tweet_id:1539382664746020864

retweets_of_tweet_id:

 Essential

_Available alias: _retweets_of_status_id:

Deliver only explicit (or native) Retweets of the specified Post. Note that the status ID used should be the ID of an original Post and not a Retweet. 

Example: retweets_of_tweet_id:1539382664746020864

​

What is a disconnection?

Establishing a connection to the streaming APIs means making a very long lived HTTPS request, and parsing the response incrementally. When connecting to the filtered stream endpoint, you should form a HTTPS request and consume the resulting stream for as long as is practical. Our servers will hold the connection open indefinitely, barring server-side error, excessive client-side lag, network issues, routine server maintenance, or duplicate logins. With connections to streaming endpoints, it is likely, and should be expected, that disconnections will take place and reconnection logic built.  

​

Why a streaming connection might be disconnected

Your stream can disconnect for a number of reasons. Inspect the error message returned by the stream to understand the reason for the failure. Possible reasons for disconnections are as follows:

• An authentication error (such as a wrong token or a wrong authentication method being used).
• A streaming server is restarted on the X side. This is usually related to a code deploy and should be generally expected and designed around.
• Your client is not keeping up with the volume of Posts the stream is delivering or is reading data too slowly. Every streaming connection is backed by a queue of messages to be sent to the client. If this queue grows too large over time, the connection will be closed.
• Your account exceeded your daily/monthly quota of Posts.
• You have too many active redundant connections.
• A client stops reading data suddenly. If the rate of Posts being read off of the stream drops suddenly, the connection will be closed.
• Possible networking issues between server and client
• A temporary server side issue, scheduled maintenance and updates. (Check the status page)  

​

Common disconnection errors include: 

	"errors": [{
		"title": "operational-disconnect",
		"disconnect_type": "UpstreamOperationalDisconnect",
		"detail": "This stream has been disconnected upstream for operational reasons.",
		"type": "https://api.x.com/2/problems/operational-disconnect"
	}]
}

{
	"title": "ConnectionException",
	"detail": "This stream is currently at the maximum allowed connection limit.",
	"connection_issue": "TooManyConnections",
	"type": "https://api.x.com/2/problems/streaming-connection"
}

​

Anticipating disconnects and reconnecting

When streaming Posts, the goal is to stay connected for as long as possible, recognizing that disconnects may occur. The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character). Use this signal to detect if you’re being disconnected.

1. Your code should detect when fresh content and the heartbeat stop arriving.
2. If that happens, your code should trigger a reconnection logic. Some clients and languages allow you to specify a read timeout, which you can set to 20 seconds.
3. Your service should detect these disconnections and reconnect as soon as possible.

Once an established connection drops, attempt to reconnect immediately. If the reconnect fails, slow down your reconnect attempts according to the type of error experienced:

• Back off linearly for TCP/IP level network errors. These problems are generally temporary and tend to clear quickly. Increase the delay in reconnects by 250ms each attempt, up to 16 seconds.
• Back off exponentially for HTTP errors for which reconnecting would be appropriate. Start with a 5 second wait, doubling each attempt, up to 320 seconds.
• Back off exponentially for HTTP 429 errors Rate limit exceeded. Start with a 1 minute wait and double each attempt. Note that every HTTP 429 received increases the time you must wait until rate limiting will no longer be in effect for your account.  

​

Recovering lost data

If you do experience a disconnect, there are some different strategies that you can use to ensure that you receive all of the data that you might have missed. We’ve documented some key steps that you can take to recover missed data on our integration guide on recovering data.   

​

Rate limits and usage

To check connection limits response will return three headers. This is useful to understand how many times you can use the rule endpoint, and how many reconnections attempts are allowed for the streaming endpoint.

• x-rate-limit-limit indicates the number of allotted requests your client is allowed to make during the 15-minute window.

• x-rate-limit-remaining indicates the number of requests made so far in the 15-minute window.

• x-rate-limit-reset is a UNIX timestamp indicating when the 15-minute window will restart, resetting x-rate-limit-remaining to 0.

The filter stream endpoint does not currently report usage data. To check how many Posts have been delivered, your code can implement a metering logic, so that consumption can be measured and paused if needed. 

Your code that hosts the client side of the stream simply inserts incoming Posts into a first in, first out (FIFO) queue, or a similar memory structure; a separate process/thread should consume Posts from that queue to parse and prepare content for storage. With this design, you can implement a service that can scale efficiently in case incoming Post volumes changes dramatically. Conceptually, you can think of it as downloading an infinitely long file over HTTP.

​

Reconnection best practices

Test backoff strategies

A good way to test a backoff implementation is to use invalid authorization credentials and examine the reconnect attempts. A good implementation will not get any 429 responses.

Issue alerts for multiple reconnects

If a client reaches its upper threshold of its time between reconnects, it should send you notifications so you can triage the issues affecting your connection.

Handle DNS changes

Test that your client process honors the DNS Time To live (TTL). Some stacks will cache a resolved address for the duration of the process and will not pick up DNS changes within the prescribed TTL. Such aggressive caching will lead to service disruptions on your client as X shifts load between IP addresses.

User Agent

Ensure your user-agent HTTP header includes the client’s version. This will be critical in diagnosing issues on X’s end. If your environment precludes setting the user-agent field, then set an x-user-agent header.

​

How to plan for high-volume social data events

Major national and global events are often accompanied by dramatic spikes in user activity across social media platforms. Sometimes these events are known about in advance, like the Super Bowl, political elections, and New Year’s celebrations around the world. Other times, the spikes in volume are due to unexpected happenings such as natural disasters, unplanned political events, pop culture moments, or health pandemics like COVID-19.

These bursts of user activity may sometimes be short-lived (measured in seconds), or they may even be sustained over several minutes’ time. No matter their origin, it is important to consider the impact that they can have on applications consuming real-time data from X. Here are some best practices that will help your team prepare for high-volume social data events.

​

Review your current filtered stream rules

• Certain keywords can skyrocket during high volume events, such as brand mentions when a brand sponsors a major sporting event.
• Be careful to avoid any unnecessary or overly generic rules that may generate unnecessary activity volumes.
• Consider communicating with your clients prior to known high-volume events to help them plan appropriately.  

​

Stress test your application

Anticipate that burst volumes may reach 5-10x average daily consumption levels. Depending on your rule set, the increase may be much higher.

​

Understand delivery caps for connections

Flow and delivery caps are based on levels of access. This results in a static volume of delivered results for streams.

• Academic: 250 Posts/second
• Enterprise: Posts/second is set at access level

​

Optimize to stay connected

With real-time streams, staying connected is essential to avoid missing data. Your client application should be able to detect a disconnect and have logic to immediately retry its connection, using an exponential backoff if the reconnect attempt fails.  

​

Add built-in buffering on your end

Building a multi-threaded application is a key strategy for handling high-volume streams. At a high-level, a best practice for managing data streams is to have a separate thread/process that establishes the streaming connection and then writes received JSON activities to a memory structure or a buffered stream reader. This ‘light-weight’ stream processing thread is responsible for handling incoming data, which can be buffered in memory, growing and shrinking as needed. Then a different thread consumes that hash and does the ‘heavy lifting’ of parsing the JSON, preparing database writes, or whatever else your application needs to do.  

​

Global events = global time zones

The events may occur after business hours or over the weekend, so be sure that your team is prepared for spikes to occur outside your normal business hours.

​

Building a client to consume streaming data

When using a streaming endpoint, there are some general best practices to consider in order to optimize usage.    

​

Client design

When building a solution with the filter stream endpoint, you will need a client that can do the following:

1. Establish an HTTPS streaming connection to the filter stream endpoint.
2. Asynchronously send POST requests to the filter stream rules endpoint to add and delete rules from the stream.
3. Handle low data volumes – Maintain the streaming connection, detecting Post objects and keep-alive signals
4. Handle high data volumes – de-couple stream ingestion from additional processing using asynchronous processes, and ensure client side buffers are flushed regularly.
5. Manage volume consumption tracking on the client side.
6. Detect stream disconnections, evaluate and reconnect to the stream automatically.  

​

Connecting to a streaming endpoint

Establishing a connection to X API v2 streaming endpoints means making a very long lived HTTP request, and parsing the response incrementally. Conceptually, you can think of it as downloading an infinitely long file over HTTP.  Once a connection has been established, the X server will deliver Post events through the connection as long as the connection is open.  

​

Consuming data in real time

Note that the individual fields of JSON objects are not ordered, and not all fields will be present in all circumstances. Similarly, separate activities are not delivered in sorted order, and duplicate messages may be encountered. Keep in mind that over time, new message types may be added and sent through the stream.

Thus, your client must tolerate:

• Fields appearing in any order
• Unexpected or missing fields
• Non-sorted Posts
• Duplicate messages
• New arbitrary message types coming down the stream at any time

In addition to relevant Post data and requested field parameters, the following kinds of messages may be delivered on a stream connection. Note that this list may not be comprehensive—additional objects may be introduced into streams. Ensure that your parser is tolerant of unexpected message formats.  

​

Buffering 

The streaming endpoints will send data to you as quickly as it becomes available, which can result in high volumes in many cases. If the X server cannot write new data to the stream right away (for example if your client is not reading fast enough, see handling disconnections for more), it will buffer the content on its end to allow your client to catch up. However, when this buffer is full, a forced disconnect will be initiated to drop the connection, and the buffered Posts will be dropped and not resent. See below for more details.

One way to identify times where your app is falling behind is to compare the timestamp of the Posts being received with the current time, and track this over time.

Although stream backups cannot ever be completely eliminated due to potential latency and hiccups over the public internet, they can be largely eliminated through proper configuration of your app. To minimize the occurrence of backups:

• Ensure that your client is reading the stream fast enough. Typically you should not do any real processing work as you read the stream. Read the stream and hand the activity to another thread/process/data store to do your processing asynchronously.
• Ensure that your data center has inbound bandwidth sufficient to accomodate large sustained data volumes as well as significantly larger spikes (e.g. 5-10x normal volume). For filtered stream, the volume and corresponding bandwidth required on your end are wholly dependent on what Posts your rules are matching.  

​

Usage tracking and rule management

As developers expectations around what “normal” data volume should be for their streams, we do not have a general recommendation for a specific percentage decrease/increase or period of time. 

Consider monitoring your stream data volumes for unexpected deviations. A data volume decrease may be symptomatic of a different issue than a stream disconnection. In such a situation, a stream would still be receiving the keep-alive signal and probably some new activity data. However, a significantly decreased number of Posts should lead you to investigate whether there is anything causing the decrease in inbound data volume to your application or network, check the status page for any related notices.

To create such monitoring, you could track the number of new Posts you expect to see in a set amount of time. If a stream’s data volume falls far enough below the specified threshold, and does not recover within a set period of time, then alerts and notifications should be initiated. You may also want to monitor for a large increase in data volume, particularly if you are in the process of modifying rules in a filtered stream, or if an event occurs that produces a spike in Post activity.

It’s important to note that Posts delivered through filtered stream do count towards the total monthly Post volume, and you should track and adjust consumption in order to optimize.  If volume is high, consider adding a sample: operator to each of your rules to reduce from 100% matching to sample:50 or sample:25 when necessary.

Additionally, we encourage you to implement measures within your app that will alert your team if the volume passes a pre-set threshold, and to possibly introduce other measures such as automated deletion of rules that are bringing in too much data, or disconnecting from the stream completely in extreme circumstances.  

​

Responding to system messages

Keep-alive signals At least every 20 seconds, the stream will send a keep-alive signal, or heartbeat in the form of an \r\n carriage return through the open connection to prevent your client from timing out. Your client application should be tolerant of the \r\n characters in the stream.

If your client properly implements a read timeout on your HTTP library, your app will be able to rely on the HTTP protocol and your HTTP library to throw an event if no data is read within this period, and you will not need to explicitly monitor for the \r\n character.

This event will typically be an exception being thrown or some other event depending on the HTTP library used. It is highly recommended to wrap your HTTP methods with error/event handlers to detect these timeouts. On timeout, your application should attempt to reconnect.

Error messages The v2 streaming endpoints may also deliver in stream error messages. Provided below is the basic format of these messages, along with some examples. Please note that the messages delivered could change, with new message being introduced. Client applications need to be tolerant of changing system message payloads.

Note that error messages will link to the documentation describing how to solve the problem.

Message format:

	"errors": [{
		"title": "operational-disconnect",
		"disconnect_type": "UpstreamOperationalDisconnect",
		"detail": "This stream has been disconnected upstream for operational reasons.",
		"type": "https://api.x.com/2/problems/operational-disconnect"
	}]
}

Note that error messages indicating a force disconnect for a full buffer may never get to your client, if the backup which caused the force disconnect prevents it from getting through. Accordingly, your app should not depend on these messages to initiate a reconnect.

​

Matching returned Posts to their associated rule

The filtered stream endpoint gives you the ability to have multiple rules in place through a single connection. Before you start receiving Posts in your stream, you’ll need to create a rule which specifies what types of Posts you’re interested in. 

When you specify your rules to match Posts based on a wide variety of attributes, including user attributes, geo-location, language, and many others, you can attach a tag to distinguish this rule at a higher level. Tags are a good way to contextualize your rule, especially if you have many rules in place. The filtering rules determine which Post activities will be sent through the connection.

If you need to add a new filtering rule to capture a different type of Post, or remove an existing rule, your app can send a request to the POST tweets/search/stream/rules endpoint to make it happen. When that request is sent, the filtering rules are automatically modified and the changes simply take effect in the data stream with no need to reconnect. Most rule additions take effect  about 20 seconds or less. It’s unlikely, but depending on external factors (for example, network connectivity), it may take longer before you start receiving matching Posts. If you can’t find a rule in the list rule endpoint, make sure that the rule creation request succeeded; this can be done by checking your logs for any error messages.

​

Matching rules

When an activity is delivered through your filtered stream connection, in a matching_rules array, it contains which list of filters matched against the Post delivered.

In the Post payload there is additional string metadata which includes the rule id and tag that caused a specific Post to be delivered. If multiple rules match a single Post, the activity is delivered a single time with each of the matching rules included in this metadata. The matching rules provide an easy way to associate a specific Post with specific rules, which is especially helpful if you have many distinct rules. Since the data is delivered through a single stream in this manner, ensuring you have unique id and tag is essential for matching. 

Here is an example of how the ”matching_rules” array appears in the Post payload:  

    {
      "id": "1166916266197536768",
      "tag": "test-rule-tag-00000"
    },
    {
      "id": "1166916266197536769",
      "tag": "test-rule-tag-12345"
    }
  ]

​

Rule tags

At the time they are created, each filtering rule may be created with a tag. Rule tags have no special meaning as they are simply treated as opaque strings carried along with a rule. They will be included in the matching_rules metadata in activities returned, and are aimed at making distinguishing your rules easier at a higher level. 

Tags provide an easy way to create logical groupings of filtering rules. For example, you may generate a unique ID for a specific rule as its tag, and allow your app to reference that ID within activities it processes to associate a result with specific customers, campaigns, categories, or other related groups.

Note that tags cannot be updated on an existing rule and can only be included when a rule is created. In order to “update” or “rename” a tag, you need to first delete the rule, then add it again with the desired tag. The best solution is to simply use a unique identifier as your tag, which your system can associate with various other data points within your own app, all without having to change anything in the rule set.

​

Filtered stream - Recovery and redundancy features

​

Introduction

When consuming realtime data, maximizing your connection time and receiving all matched data is a fundamental goal. This means that it is important to take advantage of redundant connections, automatically detect disconnections, to reconnect quickly, and to have a plan for recovering lost data.

In this integration guide, we will discuss different recovery and redundancy features: redundant connections, backfill, and recovery.  

Redundant connections

A redundant connection simply allows you to establish more than one simultaneous connections to the filtered stream. This provides redundancy by allowing you to connect to the same stream with two separate consumers, receiving the same data through both connections. Thus, your app has a hot failover for various situations such as if one stream is disconnected or if your application’s primary server fails.

Filtered stream currently only allows Projects with Enterprise access to connect to up to two redundant connections. To use a redundant stream, simply connect to the same URL used for your primary connection. The data for your stream will be sent through both connections.

​

Recovering missed data after a disconnection: Backfill

After you’ve detected a disconnection, your system should be smart enough to reconnect to the stream. If possible, your system should take note of how long the disconnection lasted so that you can use the proper recovery feature to backfill the data. 

If you are using a Project with Enterprise access and identified that the disconnection lasted five minutes or less, you can use the backfill parameter, backfill_minutes. If you pass this parameter with your GET /tweets/search/stream request, you will receive the Posts that match your rules within the past one to five minutes. We generally deliver these older Posts first before any newly matched Posts, and also do not deduplicate Posts. This means that if you were disconnected for 90 seconds, but request two minutes worth of backfill data, you will receive 30 seconds worth of duplicate Posts, which your system should be tolerant of. Here is an example of what a request might look like with the backfill parameter:

curl 'https://api.x.com/2/tweets/search/stream?backfill_minutes=5' -H "Authorization: Bearer $ACCESS_TOKEN"

If you don’t have Enterprise access, or identified that the disconnection time lasted for longer than five minutes, you can utilize the recent search endpoint or the recovery feature to request missed data. However, note that the search Posts endpoints do not include the sample:, bio:, bio_name:, or bio_location: operators, and has certain differences in matching behavior when using accents and diacritics with the keyword and #hashtag operators. These differences could mean that you don’t fully recover all Posts that might have been received via the filtered stream endpoints. 

Recovering missed data after a disconnection: Recovery

If you are using a Project with Enterprise access, you can use the Recovery feature to recover missed data within the last 24 hours if you are unable to reconnect with the 5 minute backfill window.

The streaming recovery feature allows you to have an extended backfill window of 24 hours. Recovery enables you to ‘replay’ the time period of missed data. A recovery stream is started when you make a connection request using ‘start_time’ and ‘end_time’ request parameters. Once connected, Recovery will re-stream the time period indicated, then disconnect.  

You will be able to make 2 concurrent requests to recovery at the same time, i.e. “two recovery jobs”. Recovery works technically in the same way as backfill, except a start and end time is defined. A recovery period is for a single time range.

Example request URL: https://api.x.com/2/tweets/search/stream?start_time=2022-07-12T15:10:00Z&end_time=2022-07-12T15:20:00Z

​

Comparing X API’s filtered stream endpoints

The v2 filtered stream endpoints group is replacing the standard v1.1 statuses/filter and PowerTrack API. If you have code, apps, or tools that use an older version of the filtered stream endpoint, and are considering migrating to the newer X API v2 endpoint, then this comparison can help you get started. 

See our more in depth migration guides for:

Migrating from Standard v1.1 compared to X API v2

Migrating from PowerTrack API migration to X API v2

The following table compares the filtered streaming endpoints X offers:  

Other migration resources

Standard v1.1 migration to X API v2

PowerTrack API migration to X API v2

X API migration hub

Check out some sample code for these endpoints

​

Standard v1.1 compared to X API v2

If you have been working with the v1.1 statuses/filter endpoint, this guide can help you understand the similarities and differences between the standard and X API v2 filtered stream endpoints.

• Similarities
  • Request parameters and operators
  • Support for Post edit history and metadata
• Differences
  • Endpoint URLs
  • App and Project requirement
  • Authentication method
  • Rule volume and persistent stream
  • Response data format
  • Request parameters
  • Availability of recovery and redundancy features
  • Query operators

​

Similarities

Request parameters and operators

The standard v1.1 statuses/filter endpoint features a few parameters that can be passed along with the request to filter the stream. With v2 filtered stream, you instead use a set of operators that can be connected together using boolean logic to filter for desired Posts. The available operators include some that are direct replacements for the existing standard v1.1 parameters. 

The following standard v1.1 request parameters have equivalent operators in X API v2:  

Support for Post edit history and metadata

Both versions provide metadata that describes any edit history. Check out the filtered stream API References and the Post edits fundamentals page for more details. 

​

Differences

Endpoint URLs

• Standard v1.1 endpoints:
  • https://stream.twitter.com/1.1/statuses/filter.json
• X API v2 endpoint:
  • https://api.x.com/2/tweets/search/stream
  • https://api.x.com/2/tweets/search/stream/rule

App and Project requirements

The X API v2 endpoints require that you use credentials from a developer App that is associated to a Project when authenticating your requests. All X API v1.1 endpoints can use credentials from standalone Apps or Apps associated with a Project.

Authentication method

The standard endpoint supports OAuth 1.0a User Context, whereas the X API v2 filtered stream endpoints supports OAuth 2.0 App-Only (also referred to as Application-only authentication). To make requests to the X API v2 version you must use a App Access Token to authenticate your requests.

If you no longer have the App Access Token that was presented to you when you created your Project and app in the developer portal, you can generate a new one by navigating to your app’s “Keys and tokens” page on the developer portal. If you’d like to generate an App Access Token programmatically, see this OAuth 2.0 App-Only guide. 

Rule volume and persistent stream

The standard v1.1 endpoint supports a single rule to filter the streaming connection. To change the rule, you have to disconnect the stream and submit a new request with the revised filtering rules submitted as parameters. 

The X API v2 filtered stream endpoint allows you to apply multiple rules to a single stream and add and remove rules to your stream while maintaining the stream connection. 

Response data format

One of the biggest differences between standard v1.1 and X API v2 endpoint versions is how you select which fields return in your payload.

For the standard endpoints, you receive many of the response fields by default, and then have the option to use parameters to identify which fields or sets of fields should return in the payload.

The X API v2 version only delivers the Post id and text fields by default. To request any additional fields or objects, you wil need to use the fields and expansions parameters. Any Post fields that you request from these endpoints will return in the primary Post object. Any expanded user, media, poll, or place objects and fields will return in an includes object within your response. You can then match any expanded objects back to the Post object by matching the IDs located in both the Post and the expanded object. 

We encourage you to read more about these new parameters in their respective guides, or by reading our guide on how to use fields and expansions. 

We have also put together a data format migration guide which can help you map standard v1.1 fields to the newer v2 fields. This guide will also provide you the specific expansion and field parameter that you will need to pass with your v2 request to return specific fields.   

In addition to the changes in how you request certain fields, X API v2 is also introducing new JSON designs for the objects returned by the APIs, including Post and user objects.

• At the JSON root level, the standard endpoints return Post objects in a statuses array, while X API v2 returns a data array. 
• Instead of referring to Retweeted and Quoted “statuses”, X API v2 JSON refers to Retweeted and Quoted Tweets. Many legacy and deprecated fields, such as contributors and user.translator_type are being removed. 
• Instead of using both favorites (in Post object) and favourites (in user object), X API v2 uses the term like. 
• X is adopting the convention that JSON values with no value (for example, null) are not written to the payload. Post and user attributes are only included if they have a non-null values.   

We also introduced a new set of fields to the Post object including the following:

• A conversation_id field
• Two new annotations fields, including context and entities
• Several new metrics fields 
• A new reply_setting field, which shows you who can reply to a given Post

Request parameters

There are also a set of standard filtered stream request parameters not supported in X API v2:

Availability of recovery and redundancy features

The X API v2 version of filtered stream introduces recovery and redundancy features that can help you maximize streaming up-time as well as recover any Posts that might have been missed due to a disconnection that lasted five minutes or less.

Redundant connections allows you to connect to a given stream up to two times, which can help ensure that you maintain a connection to the stream at all times, even if one of your connections fails. 

The backfill_minutes parameter can be used to recover up to five minutes of missed data. 

Both of these features are only available via Academic Research access. You can learn more about this functionality via our recovery and redundancy features integration guide. 

New query operators

X API v2 introduces new operators in support of two new features: 

• Conversation IDs - As conversations unfold on X, a conservation ID will be available to mark Posts that are part of the conversation. All Posts in the conversation will have their conversation_id set to the Post ID that started it. 
  • conversation_id:
• X Annotations provide contextual information about Posts, and include entity and context annotations. Entities are comprised of people, places, products and organizations. Contexts are domains, or topics, that surfaced entities are a part of. For example, people mentioned in a Post may have a context that indicates whether they are an athlete, actor, or politician.  
  • context: - matches on Posts that have been annotated with a context of interest. 
  • entity: - matches on Posts that have been annotated with an entity of interest. 

​

PowerTrack API migration to X API v2 filtered stream

Use this migration guide to understand the similarities and differences between PowerTrack API and X API v2 filtered stream, and to help migrate a current PowerTrack API integration to v2 filtered stream.

• Similarities
  • Streaming delivery method
  • Integration process
  • Persistent stream connection with separate rules management endpoints
  • Rule syntax
  • Rule operators (with exceptions)
  • Rule matching logic
  • Support for Post edit history and metadata
• Differences
  • Rule length
  • Rule volume
  • Endpoint URLs
  • App and Project requirement for access
  • Authentication method
  • Request parameters
  • Usage tracking
  • Multiple streams, redundant conections, backfill and Replay recovery
  • Request parameters and response format
  • Response JSON data structure

​

Similarities

Streaming delivery method

Both PowerTrack and X API v2 filtered stream use streaming data delivery, which require the client to establish an open connection to an endpoint and keeping a very long lived HTTP request, and parsing the response incrementally from the server in real time.  Both PowerTrack and X API v2 filtered stream filter publicly available Posts matching rules that exist on the stream in real time, and use keep-alive signals as new line characters (\r\n) to signal the connection is still active. Both PowerTrack and X API v2 filtered stream endpoint connections deliver data in real time and should be read by the connecting client quickly.  

Integration process

Integrating with filtered stream is similar to integrating with PowerTrack, using the general process below:

1. Establish a streaming connection.
2. Asynchronously send separate requests to add and delete rules from the stream.
3. Reconnect to the stream automatically when connection is disconnected.

Persistent stream connection with separate rules management endpoints

Similar to the PowerTrack API and Rules API, the new X API v2 filtered stream endpoints allows you to apply multiple rules to a single stream and add and remove rules to your stream while maintaining the stream connection.

Rule syntax, operators, and matching rules logic

The X API v2 filtered stream uses a subset of the same rule operators currently used for PowerTrack rules. These operators are used to create boolean based rule syntax used for filtering desired matching Posts from the live stream.  Both PowerTrack and filtered stream use the same rule syntax for building rules and matching logic is the same. While the majority of the operators are available for both PowerTrack and filter stream, there are a few notable differences and net new operators listed below.  For more details and example uses for each operator see current PowerTrack operators and current X API v2 filtered stream operators. 

Please note that many operators (noted as ‘advanced operators’) are reserved for those users who have been approved for Academic Research access or Enterprise access.

Operators available with both PowerTrack and X API v2 Filtered stream:

Support for Post edit history and metadata

Both versions provide metadata that describes any edit history. Check out the filtered stream API References and the Edit Posts fundamentals page for more details. 

​

Differences

Rule length

Rule length is measured the same way (by character count) for both PowerTrack and filtered stream rules, however the maximum length for PowerTrack rules is 2048 characters and the maximum rule length for rules on X API v2 filtered stream varies by access level. 

Enterprise access -  2048 characters (please contact your designated account manager regarding your specific account)

Rule volume

The PowerTrack maximum rule volume per stream is defined within the enterprise account contract.  X API v2 filtered stream rule volume varies by access level.

Enterprise access - 25000+ rules (please contact your designated account manager regarding your specific account)  

Endpoint URLs

• PowerTrack endpoints:
  • https://gnip-stream.twitter.com/stream/powertrack/accounts/{account\_name}/publishers/twitter/{stream\_label}.json
  • https://gnip-api.x.com/rules/powertrack/accounts/{account\_name}/publishers/twitter/{stream\_label}.json
  • https://gnip-api.x.com/rules/powertrack/accounts/{account\_name}/publishers/twitter/{stream\_label}/validation.json
• X API v2 endpoint:
  • https://api.x.com/2/tweets/search/stream
  • https://api.x.com/2/tweets/search/stream/rule

App and Project requirements for v2 access

PowerTrack access is granted through a contracted annual subscription for data, and set up through console.gnip.com by your account manager at X.  PowerTrack does not require a X developer App to access.  In order to use the X API v2 filter stream, you must have signed up for a X developer account, and a X developer App associated with a Project. The developer App and Project setup for X API v2 access is all done through the developer portal.  

Authentication method

The PowerTrack API endpoints use Basic Authentication set up in console.gnip.com. The X API v2 filtered stream endpoints require a X developer App and an OAuth 2.0 App Access Token (also referred to as Application-only or Bearer Authentication). To make requests to the X API v2 version you must use your specific developer App’s Access Token to authenticate your requests.

In the process of setting up your developer account, developer App and Project, an App Access Token is created and shared within the dev portal user interface, however, you can generate a new one by navigating to your app’s “Keys and tokens” page on the developer portal. If you’d like to generate/destroy the App Access Tokens programmatically, see this OAuth 2.0 App-Only guide.

Usage tracking

PowerTrack usage can be retrieved programatically using the Usage API, or can be seen in console.gnip.com on the usage tab.  Post consumption across all PowerTrack streams is deduplicated each day and volume consumption is defined within the enterprise contract. 

X API v2 filtered stream usage can be tracked within the developer portal at the Project level. Post consumption is set at the Project level and is shared across several different X API v2 endpoints, including filtered stream, recent search, user Post timeline and user mention timeline.  Currently with Basic Access, the monthly Post consumption limit is 500,000 Posts per month total.

Multiple streams, redundant conections, backfill and Replay API for recovery

There are several recovery and redunancy features available via PowerTrack, some of which are not yet available for X API v2 filtered stream.  For PowerTrack, all recovery and redundancy features are configured within the enterprise contract. PowerTrack API currently has the flexibility to offer multiple PowerTrack streams (commonly “dev” and “production”) with unique rulesets.  Currently, the X API v2 filtered stream is only available with a single stream.

PowerTrack allows you to connect to have multiple connections to a single stream, generally used for redundant connections to different data centers or clients. 

If a PowerTrack stream is disconnected breifly, a reconnection can be made using the backfillMinutes parameter to reduce the chance of data loss within five minutes of the disconection. While we have added this functionality to the X API v2 version, it is currently only available with Academic Research access, and has been renamed to backfill_minutes.

If a PowerTrack stream is disconnected for longer than a 5 minute period, the separate Replay API can be used to recover data for up to a 2 hour period in the recent 5 day past.  Currently, the X API v2 filtered stream does not have a replay feature.

Request parameters and response format

One of the biggest differences between PowerTrack API and X API v2 filtered stream is the parameter functionality.

Using the X API v2 filtered stream, there are several parameters used on the connection request to identify which fields or expanded objects to return in the Post payload.  This is common for all v2 endpoints. By default, only the Post id and text are returned for matching Posts but additional parameters, fields and expansions described below, can be added in order to recieve more detailed data per matching Post. 

fields: X API v2 endpoints enable you to select which fields are provided in your payload. For example, Post, User, Media, Place, and Poll objects all have a list of fields that can be returned (or not). 

expansions: Used to expand the complementary objects referenced in Post JSON payloads. For example, all Retweets and Replies reference other Posts. By setting expansions=referenced_tweets.id, these other Post objects are expanded according to the `tweet.fields` setting.  Other objects such as users, polls, and media can be expanded.

https://api.x.com/2/tweets/search/stream?tweet.fields=created\_at,public\_metrics,author\_id,entities&expansions=attachments.poll\_ids

You can learn more about how to use fields and expansions by visiting our guide, and by reading through our broader data dictionary.

The PowerTrack API data format is set within console.gnip.com at the stream settings level, which can be set to either the X native enriched format or Activity streams format. 

PowerTrack API only uses one optional parameter on connection, to reconnect using backfill (backfillMinutes=5). This optional parameter is also available to filtered stream, but is called backfill_minutes, and is only available via Academic Research access.  

https://gnip-stream.twitter.com/stream/powertrack/accounts/{account\_name}/publishers/twitter/{stream\_label}.json?backfillMinutes=5

Response structure and data format

As described above, the request parameters set at the connection request for X API v2 filtered stream determine the response data returned.  There are several different response possibilites using different fields and expansions which can range from the most simple default response with only the Post id and text, to an extremely detailed and expanded data payload.

The data format for PowerTrack is set within console.gnip.com at the stream settings level, which can be set to either the X Native Enriched format or Activity Streams format. 

The following table references Post response examples in each different format:

If you would like to know more about how the enterprise data formats map to the X API v2 format, please visit our following guides:

• Native Enriched to X API v2 format migration guide
• Activity Streams to X API v2 format migration guide

​

API reference index

For the complete API reference, select an endpoint from the list:

​

GET /2/tweets/search/stream/rules

Return either a single rule, or a list of rules that have been added to the stream.

If you would like to initiate the stream to receive all Tweets that match these rules in real-time, you will need to use the GET /tweets/search/stream endpoint.

​

Endpoint URL

https://api.x.com/2/tweets/search/stream/rules

​

Authentication and rate limits

​

Query parameters

​

Example code with offical SDKs

• TypeScript
• Java

(async () => {
  try {
    const getStreamRules = await twitterClient.tweets.getRules();
    console.dir(getStreamRules, {
      depth: null,
    });
  } catch (error) {
    console.log(error);
  }
})();

​

Example responses

• Response

  "data": [
    {
      "id": "1165037377523306497",
      "value": "dog has:images",
      "tag": "dog pictures"
    },
    {
      "id": "1165037377523306498",
      "value": "cat has:images -grumpy"
    }
  ],
  "meta": {
    "sent": "2019-08-29T01:12:10.729Z"
  }
}

​

Response fields

​

GET /2/tweets/search/stream

Streams Tweets in real-time that match the rules that you added to the stream using the POST /tweets/search/stream/rules endpoint. If you haven’t added any rules to your stream, you will not receive any Tweets.

If you have been approved for Academic Research access, you can take advantage of the redundant connections functionality, which allows you to connect to a stream up to two times, which will help maximize your streaming up-time.

The Tweets returned by this endpoint count towards the Project-level Tweet cap.

Run in Postman ❯

Build request with API Explorer ❯

​

Endpoint URL

https://api.x.com/2/tweets/search/stream

​

Authentication and rate limits

​

Query parameters

​

Example code with offical SDKs

• TypeScript (Default fields)
• TypeScript (Optional fields)
• Java (Default fields)
• Java (Optional fields)

(async () => {
  try {
    const searchFilteredStream = await twitterClient.tweets.searchStream();
    console.dir(searchFilteredStream, {
      depth: null,
    });
  } catch (error) {
    console.log(error);
  }
})();

​

Example responses

• Default format

{
  "data": {
    "id": "1067094924124872705",
    "edit_history_tweet_ids": [
      "1067094924124872705"
    ],
    "text": "Just getting started with Twitter APIs? Find out what you need in order to build an app. Watch this video! https://t.co/Hg8nkfoizN"
  }
}

​

Response fields

Filtered stream

​

POST /2/tweets/search/stream/rules

Add or delete rules to your stream.

Once you’ve added a rule or rules to your stream, you can retrieve all of the Tweets that match these rules by using the GET /tweets/search/stream endpoint.

To learn how to build a rule, please read our guide on building a rule.

To create one or more rules, submit an add JSON body with an array of rules and operators. Similarly, to delete one or more rules, submit a delete JSON body with an array of list of existing rule IDs.

Run in Postman ❯

Build request with API Explorer ❯

​

Endpoint URL

https://api.x.com/2/tweets/search/stream/rules

​

Authentication and rate limits

​

Query parameters

​

JSON body parameters

​

Example code with offical SDKs

• TypeScript
• Java

(async () => {
  try {
    const addOrDelete = await twitterClient.tweets.addOrDeleteRules(
      {
        add: [
          {
            //The value of the rule text
            value: "cat has:media",
            //A tag meant for the labeling of user provided rules.
            tag: "cats with media",
          },
          {
            value: "cat has:media -grumpy",
            tag: "happy cats with media",
          },
        ],

        //To delete rules comment out the add section and uncomment out the delete code
        /*
        delete: {
          //IDs of all deleted user-specified stream filtering rules.
          ids: ["1165037377523306498", "1165037377523306499"],
        },
        */
      },
      //Optional - Dry Run can be used with both the add and delete action, with the expected result given, but without actually taking any action in the system (meaning the end state will always be as it was when the request was submitted). This is particularly useful to validate rule changes.
      { dry_run: true }
    );
    console.dir(addOrDelete, {
      depth: null,
    });
  } catch (error) {
    console.log(error);
  }
})();

​

Example responses

• Create rules
• Success

{
  "data": [
    {
      "value": "meme",
      "tag": "funny things",
      "id": "1166895166390583299"
    },
    {
      "value": "cats has:media -grumpy",
      "tag": "happy cats with media",
      "id": "1166895166390583296"
    },
    {
      "value": "cat has:media",
      "tag": "cats with media",
      "id": "1166895166390583297"
    },
    {
      "value": "meme has:images",
      "id": "1166895166390583298"
    }
  ],
  "meta": {
    "sent": "2019-08-29T02:07:42.205Z",
    "summary": {
      "created": 4,
      "not_created": 0
    }
  }
}

​

Response fields