Compliance Firehose API

Please note We have released a new compliance tool to X API v2 called batch compliance. This new tool allows you to upload large datasets of Post or user IDs to retrieve their compliance status in order to determine what data requires action in order to bring your datasets into compliance.

In addtion, both the batch compliance and the compliance firehose have been updated to support Post edits. For the compliance firehose, a new ‘tweet_edit’ event was added. See the Compliance Data Objects documentation for more details. Learn more about how Edit Post metadata works on the Edit Posts fundamentals page.

​

Overview

​

GET statuses/lookup and GET users/lookup

• Format: REST API’s See: GET statuses/lookup and GET users/lookup.
• These endpoints always return the latest version of any Post edits. All Post objects describing Posts created after the edit feature was introduced will include Post edit metadata. This is true even for Posts that were not edited. 
• For all Posts, requests for Posts more than 30 minutes after they were created will represent the final state of all Posts. 
• Deliver availability information for specific Posts or Users as defined by the caller as part of the API request.
• May be used for ad-hoc spot-checking on the current availability state of a specific group of Posts / Users.
• Ideal for customers who need a way to check the current state of a specific Post or User at a given moment in time.
• These API’s provide a helpful mechanism that may be used by customers who need to check the availability of a piece of Content, for instance when:
  1. Displaying Posts
  2. Engaging with a Post(s) or User(s) in a 1:1 way
  3. Distributing X Content to a 3rd party through an allowed file download
  4. Storing Posts for extended periods of time

​

Compliance Firehose (enterprise only)

• Format: Streaming API See: Compliance Firehose.
• Delivers realtime stream of Compliance activities on X. These activities include when Posts are edited. 
• May be used to maintain compliance state across a set of stored data as new compliance events happen on the platform.
• Ideal for customers consuming and storing large quantities of X data for extended periods of time.

​

Guides

​

Compliance Best Practices

​

Recommendations & Best Practices

• Build Data Storage Schemas That Store Numeric Post ID and User ID: User messages require action to be taken on all Posts from that User. Therefore, since all compliance messages are delivered only by numeric ID, it is important to design storage schemas that maintain the relationship between Post and User based on numeric IDs. Data consumers will need to monitor compliance events by both Post ID and User ID and be able to update the local data store appropriately.
• Build Schemas That Address All Compliance Statuses: Depending on how compliance activities will be addressed in various applications, it may be required to add other metadata to the data store. For instance, data consumers may decide to add metadata to an existing database to facilitate restricting the display of content in countries affected by a status_withheld message.
• Handling Retweet Deletes: Retweets are a special kind of Post where the original message is nested in an object within the Retweet. In this case, there are two Post IDs referenced in a Retweet – the ID for the Retweet, and the ID for the original message (included in the nested object). When an original message is deleted, a Post delete message is issued for the original ID. Post deletion events typically trigger delete events for all Retweets. However, in some cases not all are sent and client systems should be tolerant of incomplete Retweet deletions. The deletion of the original ID should be sufficient to delete all subsequent Retweets. It is a best practice to reference the original Post ID when storing Retweets, and deleting all referenced Retweets when receiving Post delete events.

​

Compliance Data Objects

​

Compliance Firehose API

• Read more about User statuses here and our developer policy around deleted Posts here.
• The Compliance Firehose has been updated to provide ‘tweet_edit’ events. 
• Several User delete, protect and suspend events are not necessarily permanent and can toggle between states infinitely. These include: user_delete,user_undelete, user_protect, user_unprotect and user_suspend, user_unsuspend.
• User_deletes are followed by status_deletes 30 days later only if the user has not selected to user_undelete their account. It is possible that a user_delete is reversed by the user and deletes for all of their Posts 30 days later do not occur.
• User_suspend is an action that remains true unless the user is subject to an user_unsuspend event. These are not subject to any changes on a 30 day time period.

​

Payload examples

{"tweet_edit":
   {
     "id": "1557445923210514432"
     "initial_tweet_id": "1557433858676740098",
     "edit_tweet_ids": ["1557433858676740098", "1557445923210514432"],
     "timestamp_ms": "1660155761384"
   }
 }

Post delete

{
  "delete": {
    "status": {
      "id": 601430178305220600,
      "id_str": "601430178305220608",
      "user_id": 3198576760,
      "user_id_str": "3198576760"
    },
    "timestamp_ms": "1432228155593"
  }
}

Post withheld

{
  "status_withheld": {
    "status": {
      "id": 601430178305220600,
      "id_str": "601430178305220608",
      "user_id": 3198576760,
      "user_id_str": "3198576760"
    },
    "withheld_in_countries": [
      "XY"
    ],
    "timestamp_ms": "1432228155593"
  }
}

Drop

{
  "drop": {
    "status": {
      "id": 601430178305220600,
      "id_str": "601430178305220600",
      "user_id": 3198576760,
      "user_id_str": "3198576760"
    },
    "timestamp_ms": "1432228155593"
  }
}

Undrop

{
  "undrop": {
    "status": {
      "id": 601430178305220600,
      "id_str": "601430178305220600",
      "user_id": 3198576760,
      "user_id_str": "3198576760"
    },
    "timestamp_ms": "1432228155593"
  }
}

​

Scrub geo

{
  "scrub_geo": {
    "user_id": 519761961,
    "up_to_status_id": 411552403083628540,
    "up_to_status_id_str": "411552403083628544",
    "user_id_str": "519761961",
    "timestamp_ms": "1432228180345"
  }
}

User delete

{
  "user_delete": {
    "id": 771136850,
    "timestamp_ms": "1432228153548"
  }
}

User undelete

{
  "user_undelete": {
    "id": 796250066,
    "timestamp_ms": "1432228149062"
  }
}

User withheld

{
  "user_withheld": {
    "user": {
      "id": 1375036644,
      "id_str": "1375036644"
    },
    "withheld_in_countries": [
      "XY"
    ],
    "timestampMs": "2014-08-27T23:49:41.839+00:00"
  }
}

User protect

{
  "user_protect": {
    "id": 3182003550,
    "timestamp_ms": "1432228177137"
  }
}

User unprotect

{
  "user_unprotect": {
    "id": 2911076065,
    "timestamp_ms": "1432228180113"
  }
}

User suspend

{
  "user_suspend": {
    "id": 3120539094,
    "timestamp_ms": "1432228194217"
  }
}

User unsuspend

{
  "user_unsuspend": {
    "id": 3293130873,
    "timestamp_ms": "1432228193828"
  }
}

​

integrating Compliance Firehose

​

Integrating with the Compliance Firehose

1. Establish a streaming connection to each streaming API partition of the Compliance Firehose
2. Handle high data volumes – de-couple stream ingestion from additional processing using asynchronous processes
3. Reconnect to the stream partitions automatically when disconnected for any reason
4. Process compliance events that are relevant to Post and User data you have stored in accordance with the guidance presented above

​

Honoring user intent on Twitter

​

User

​

Status

​

Keeping Track of User and Status Changes

​

API Reference

​

GET compliance/firehose

​

Methods

​

Authentication

​

GET /compliance/:stream

curl --compressed -v -uexample@customer.com "https://gnip-stream.x.com/stream/compliance/accounts/:account_name/publishers/twitter/:stream_label.json?partition=1"

​

Response Codes

​

Other Recommendations & Best practices

• Build Data Storage Schemas That Store Numeric Tweet ID and User ID: User messages require action to be taken on all Tweets from that User. Therefore, since all compliance messages are delivered only by numeric ID, it is important to design storage schemas that maintain the relationship between Tweet and User based on numeric IDs. Data consumers will need to monitor compliance events by both Tweet ID and User ID and be able to update the local data store appropriately.
• Build Schemas That Address All Compliance Statuses: Depending on how compliance activities will be addressed in various applications, it may be required to add other metadata to the data store. For instance, data consumers may decide to add metadata to an existing database to facilitate restricting the display of content in countries affected by a status_withheld message.
• Handling Retweet Deletes: Retweets are a special kind of Tweet where the original message is nested in an object within the Retweet. In this case, there are two Tweet IDs referenced in a Retweet — the ID for the Retweet, and the ID for the original message (included in the nested object). When an original message is deleted, a Tweet delete message is issued for the original ID. Subsequent delete messages are NOT issued for all of the Retweets. The deletion of the original ID should be sufficient to delete all subsequent Retweets.