How to Hydrate Tweets: Turn a Twitter (X) Tweet ID Dataset Back Into Full Tweets

- Hydrating = sending tweet IDs back to an API and getting the full tweet JSON in return. Public datasets ship as ID lists because X's developer policy allows sharing Post IDs, not the posts.
- twarc and Hydrator need official X API keys. twarc's README says it is no longer actively supported; Hydrator's README says its application keys were rescinded by Twitter.
- Two ways today: X's
GET /2/tweets(up to 100 IDs per call, $0.005 per post read) or twitterapi.io/twitter/tweets(up to 50 IDs per request, $0.15 per 1,000 tweets returned). We confirmed the 50 limit: 51 IDs returned HTTP 400. - Only 53.9% of 1,900 sampled IDs still hydrated, from a COVID-19 tweet-ID dataset of 2020 to early 2023 posts, collected in 2026. By tweet year: 2020 45.5%, 2021 57.0%, 2022 57.2%, early 2023 67.0%.
- Cost per 100,000 IDs: about $8.09 on twitterapi.io at that survival rate (2,000 requests, ~53,947 tweets returned), $15 if every ID survived, versus $500 for 100,000 post reads on X's rate card.
- Match results by
id, never by position: the API returns only the tweets that still exist, in its own order, and the response does not say why an ID is missing.
Most public Twitter (X) research datasets are not tweets at all. They are text files of long numbers, one tweet ID per line, sometimes hundreds of millions of them. To use one, you have to hydrate it: look every ID up again and rebuild the tweets. For a decade the standard answer was twarc or the DocNow Hydrator app with free academic API access. That answer stopped working when X moved its API to paid access, and most tutorials that rank for this topic still describe the old route.
This guide is the hydration step specifically. If you are designing your own collection, read the dataset export guide and the academic research API guide linked below; if you want to recover one deleted tweet, see the deleted tweet search guide. Here we cover why datasets ship as IDs (with X's policy text), what happened to twarc and Hydrator, a resumable Python hydrator that batches 50 IDs per request, what it costs per 100,000 IDs against X's own rate card, and first-hand numbers on how many IDs from a well-known public dataset still come back.
53.9%
Why tweet datasets ship as IDs (the policy text)
X's Developer Policy has a section titled Content redistribution. It opens with the rule: "we restrict the redistribution of X Content to third parties." It then says that if you share data you collected, "you may only distribute Post IDs, Direct Message IDs, and/or User IDs" (with exceptions described below it). That single sentence is why almost every public tweet dataset on GitHub, Zenodo or Harvard Dataverse is a list of numbers instead of a table of tweets. The policy is published at docs.x.com/developer-terms/policy.
The volume cap. The same section says "you may not distribute more than 1,500,000 Post IDs to any entity" within any 30-day period unless X gives written permission. Large datasets, such as the COVID-19 ID collection we sample below with roughly 2.78 billion IDs, rely on the academic carve-out instead.
The academic carve-out. "Academic researchers are permitted to distribute Post IDs and/or User IDs solely for" non-commercial research on behalf of an academic institution, or for peer review and validation of that research. This is the clause that lets university labs publish very large ID lists.
Small amounts of full posts. The policy also allows developers to "provide up to 500 public Posts Objects and/or User Objects to each person who uses your service" per day, by non-automated means such as a manual download. That is why some datasets ship a few hundred full examples alongside the full ID list.
The EU research route. The section ends with: "This Section does not apply to researchers with X API access via Art. 40 of the EU Digital Services Act (2022)." Researchers with data access under DSA Article 40 work under separate terms.
The practical consequence is a deliberate design: the dataset author shares IDs, and every user rebuilds the tweets themselves at the time they use the data. Tweets that were deleted, protected or suspended since collection do not come back, which is how the redistribution rule respects a user's later decision to remove a post. It is also why two people who hydrate the same ID file a year apart get different datasets.
What happened to twarc, Hydrator and academictwitteR
Three open-source tools carried most of the hydration work in research. All three call X's own API with credentials you register, so their status depends on X's access rules, not only on their code. We checked each project's own page in this session.
twarc still installs and the command still exists. The twarc2 docs describe twarc2 hydrate [OPTIONS] [INFILE] [OUTFILE], which "will read a file of tweet identifiers and write out the tweet JSON for them", with input of "a tweet identifier on each line, without quotes or a header". What changed is the access behind it: X's API is now paid (academictwitteR's README puts it plainly: "The paid API means open-source development of this package is no longer feasible"), so any twarc run today uses a paid X developer account and is billed on X's current rate card. We did not run twarc for this page, so we make no claim about whether every twarc code path works against today's API.
Hydrator cannot be fixed by the user. It shipped with built-in app keys; with those rescinded, the desktop app has nothing to authenticate with. If you open an old guide that says "download Hydrator and drag your file in", that route is closed.
What this means for a methods section. If your paper or thesis cites a dataset that was hydrated with twarc in 2021, a replication today needs a different tool, and the survival rate will be lower than what the original authors saw. Write the tool, the date of hydration (year is enough for most journals) and the share of IDs that came back into your methods.
Option 1: hydrate through X's official API (GET /2/tweets)
X's own lookup endpoint is GET /2/tweets with an ids parameter. The API reference at docs.x.com/x-api/posts/get-posts-by-ids lists the parameter as an array with a minimum of 1 and a maximum of 100 IDs, so 100,000 IDs is 1,000 requests. Posts that cannot be returned come back in an errors array next to data, typed as problems such as resource-not-found or not-authorized-for-resource. That gives you more detail on why an ID is missing than most third-party APIs do.
Price. X's pricing page (docs.x.com/x-api/getting-started/pricing) lists post reads at $0.005 per resource and user reads at $0.010 per resource on the pay-per-usage plan. Hydrating 100,000 IDs where every post still exists is therefore $500. The page also says resources are deduplicated within a 24-hour UTC window ("Requesting the same Post multiple times in a day counts as one charge"), which X describes as a soft guarantee.
Caps and credits. Pay-per-usage is limited to 3 million post reads per monthly billing cycle; above that X points to Enterprise. New accounts that save an eligible card and buy credits get $20 in API credits, and the first auto-recharge is matched up to $50, per the same page.
When to choose it. If you need X's own error reasons per ID (for example, to separate deleted posts from protected accounts in a methods section), or your institution already has an X developer account with credits, the official endpoint is the direct route. Your bearer token goes in an Authorization: Bearer header, and the same pattern of reading an ID file in chunks applies; only the chunk size (100) and the response shape differ.
Option 2: hydrate with /twitter/tweets in batches of 50
twitterapi.io exposes tweet lookup by ID as GET https://api.twitterapi.io/twitter/tweets with a tweet_ids parameter: a comma-separated list of IDs and your key in the X-API-Key header. No X developer account or X login is needed for reads. The endpoint reference at docs.twitterapi.io/api-reference/endpoint/get_tweet_by_ids says "Maximum 50 IDs per request (more returns HTTP 400)". We checked that limit with a real request: 50 IDs returned HTTP 200 with tweet objects in 38 out of 38 batches, and 51 IDs returned HTTP 400.
What comes back. The response is {"tweets": [...], "status": "success", "msg": "success", "code": 0}. Each tweet object carries id, url, text, createdAt (for example "Thu Oct 16 12:34:56 +0000 2026"), lang, likeCount, retweetCount, replyCount, quoteCount, viewCount, bookmarkCount, isReply, inReplyToId, conversationId, entities, extendedEntities for media, the nested quoted_tweet and retweeted_tweet, and an author object with userName, name, followers, isBlueVerified and account dates. That is enough to rebuild the columns most research datasets need.
What does not come back. IDs that no longer resolve are simply absent. The docs say "IDs that are deleted, private or from suspended accounts are not returned (occasionally an entry with an empty id); match results by id." The API does not say which of those reasons applies, so with this endpoint you can report returned versus not returned, but not deleted versus suspended. In our 38 batches we saw zero entries with an empty id and zero tweets we had not asked for, and the returned tweets were not in input order. Always build a dictionary keyed by id.
Price. Tweet endpoints are billed at 15 credits per tweet returned, which is $0.15 per 1,000 tweets, with a minimum of 15 credits ($0.00015) per request. A batch where all 50 IDs come back costs 750 credits ($0.0075); a batch where 27 come back costs 405 credits. Pay only for the data returned: the price is set per tweet in the response, not per ID you send. New accounts that pass the automatic bot check get up to $1 of free credits (at most 100,000 credits, no card, one per person or network, expiring after 31 days), which covers hydrating several thousand IDs as a pilot.
Throughput. Before the first payment the rate limit is 0.2 requests per second (one batch every five seconds, about 36,000 IDs an hour). After the first payment it is 20 requests per second, 50 on the Scale plan and 500 on Business. At 20 requests per second a single process can send 1,000 IDs per second; in practice you will want a few concurrent workers and a checkpoint file, as in the script below.
A resumable hydrator: IDs in, JSONL out, missing IDs logged
Real ID files are large and hydration runs for hours, so the script has to survive interruptions. The pattern below reads the ID file, skips IDs already processed (by reading the output and the missing log), sends batches of 50, writes every returned tweet as one JSON line, and writes every ID that did not come back to missing.txt. Re-running it after a crash continues where it stopped.
Three details that matter. First, deduplicate and validate IDs before you send them: hourly dataset files sometimes repeat IDs, and a stray header line will make a whole batch fail. Second, match by id as a string; Python handles 64-bit integers, but JavaScript and spreadsheets do not, so keep IDs as strings end to end. Third, back off on HTTP 429 and retry 5xx responses; never retry an HTTP 400 blindly, because it usually means a malformed batch.
import json, os, time
import requests
BASE = "https://api.twitterapi.io"
KEY = os.environ["TWITTERAPI_IO_KEY"]
BATCH = 50 # documented maximum; 51 IDs returns HTTP 400
def load_ids(path):
seen, out = set(), []
for line in open(path):
s = line.strip()
if s.isdigit() and s not in seen:
seen.add(s)
out.append(s)
return out
def done_ids(out_path, missing_path):
done = set()
if os.path.exists(out_path):
done |= {json.loads(l)["id"] for l in open(out_path)}
if os.path.exists(missing_path):
done |= {l.strip() for l in open(missing_path)}
return done
def lookup(batch):
for attempt in range(6):
r = requests.get(f"{BASE}/twitter/tweets",
params={"tweet_ids": ",".join(batch)},
headers={"X-API-Key": KEY}, timeout=60)
if r.status_code == 200:
return r.json().get("tweets") or []
if r.status_code in (429, 500, 502, 503, 504):
time.sleep(2 ** attempt)
continue
r.raise_for_status()
raise RuntimeError("gave up on a batch")
def hydrate(id_path, out_path="tweets.jsonl", missing_path="missing.txt", pause=5.0):
todo = [i for i in load_ids(id_path) if i not in done_ids(out_path, missing_path)]
with open(out_path, "a") as out, open(missing_path, "a") as miss:
for k in range(0, len(todo), BATCH):
batch = todo[k:k + BATCH]
by_id = {str(t["id"]): t for t in lookup(batch) if t.get("id")}
for i in batch:
if i in by_id:
out.write(json.dumps(by_id[i], ensure_ascii=False) + "\n")
else:
miss.write(i + "\n")
out.flush(); miss.flush()
time.sleep(pause) # 5 s fits the 0.2 req/s limit before the first payment
hydrate("ids.txt")How many IDs still hydrate? A 1,900-ID test on a public dataset
Guides usually say "some tweets will be missing". We measured how many. We took the COVID-19-TweetIDs dataset (Chen, Lerman and Ferrara, JMIR Public Health and Surveillance, 2020), one of the most widely cited public tweet-ID collections, published at github.com/echen102/COVID-19-TweetIDs under CC BY-NC-SA 4.0. It is organized as one file per UTC hour in monthly folders from January 2020 to February 2023. For each of the 38 months we drew 50 random IDs from two hourly files and hydrated all 1,900 IDs through /twitter/tweets, collected in 2026.
Headline: 1,025 of 1,900 IDs (53.9%) came back as tweets; 875 (46.1%) did not. For context, the dataset's own README reports that about 6% of tweets were already gone when its first version was hydrated; in our sample the missing share is now 46.1%. The samples differ, so treat this as a direction, not a precise delta.
Older tweets survive less. The 2020 IDs hydrate at 45.5% against about 57% for 2021 and 2022. The weakest month was October 2020 (13 of 50, 26%), and the strongest was August 2022 (36 of 50, 72%). With 50 IDs per month the monthly bars move by several points from sampling noise alone, so read the year averages, not single months.
Most of what survives are retweets. Of the 1,025 returned tweets, 66.3% were retweets, 18.7% quoted another tweet and 11.4% were replies. The language mix was 58.7% English, 12.0% Spanish, 5.8% French and 3.4% German. Only 8.0% of the returned tweets were from authors with isBlueVerified true today. A retweet ID only hydrates if both the retweeting account and the original post still exist, which is one reason keyword-stream datasets, heavy with retweets, lose IDs faster than a set of original posts would.
What the sample cannot show. The endpoint does not tell us why an ID is missing, so we do not split the 46.1% into deleted posts, protected accounts and suspended accounts. The COVID-19 stream is one topic, collected by keyword, during a period with intense moderation of health misinformation, so other datasets will differ. It is one hydration pass at one point in time.

What hydration costs per 100,000 IDs
Because twitterapi.io prices tweet endpoints per tweet returned, the cost of a hydration job depends on how many IDs survive, not only on how many you send. The minimum of 15 credits per request matters only for batches where almost nothing comes back. Here is the arithmetic for 100,000 IDs, using the 53.9% survival rate from the study above.
The comparison on one rate card. X's pay-per-usage rate card lists $0.005 per post read, which is $5 per 1,000 posts. twitterapi.io charges $0.15 per 1,000 tweets returned. Per 1,000 tweets that is about 33 times less on twitterapi.io, and the gap holds at any survival rate as long as both sides bill only posts that come back. If X billed only the roughly 53,947 posts that would come back from our sample's survival rate, the X figure would be about $270.
Scaling up. One million IDs at the same survival rate is about $81 on twitterapi.io, and the full 1.5 million IDs that X's policy lets a non-academic dataset share in 30 days would be about $121. For very large corpora, the monthly caps and plans on /pricing matter more than the unit price; credits you purchase never expire, so you can buy for a whole project and hydrate over months.
Our own bill for this study. 1,025 tweets returned across 38 hydration batches comes to about 15,375 credits, roughly $0.15 at list price, plus one rejected 51-ID test request.

After hydration: decode dates from IDs and document the gaps
Every tweet ID carries its own timestamp. Since late 2010, tweet IDs are snowflake IDs: the top bits hold milliseconds since X's epoch. ms = (id >> 22) + 1288834974657 gives the Unix time in milliseconds, so you can bucket the missing IDs by month without any API call. That is how the chart above places IDs that no longer exist. The free Twitter timestamp tool linked below does the same conversion for a single ID.
Report the survival rate in your methods. A reviewer cannot rehydrate your exact corpus later, because more IDs will be gone. Store the hydrated JSONL with the year of hydration, the tool and endpoint, the number of IDs sent and the number returned. If a reviewer needs to verify results, share the ID list and your hydration counts, which is exactly what X's redistribution clause allows.
Check whether the gaps bias your question. Deleted and suspended content is not random. In our sample, 2020 IDs survived at 45.5% versus 57% in the following two years. If your study compares periods, or studies a topic where moderation or account suspensions were heavy, compare the time distribution of returned IDs against the full ID list before drawing conclusions. The snowflake decode makes that check cheap.
Respect the authors. Hydrated tweets belong to identifiable people. Follow your ethics board's rules on storage and anonymization, publish aggregates rather than quoted posts from private individuals, and re-share IDs rather than the hydrated JSON.
from collections import Counter
from datetime import datetime, timezone
EPOCH_MS = 1288834974657 # X snowflake epoch
def tweet_time(tweet_id: str) -> datetime:
ms = (int(tweet_id) >> 22) + EPOCH_MS
return datetime.fromtimestamp(ms / 1000, timezone.utc)
sent = [l.strip() for l in open("ids.txt") if l.strip().isdigit()]
missing = {l.strip() for l in open("missing.txt")}
total, ok = Counter(), Counter()
for i in sent:
year = tweet_time(i).year
total[year] += 1
ok[year] += i not in missing
for year in sorted(total):
print(year, total[year], f"{ok[year] / total[year]:.1%} returned")How we collected this
Question. If you hydrate a well-known public tweet-ID dataset today, what share of IDs still come back as tweets, and does it depend on the year the tweet was posted?
Dataset and sample. COVID-19-TweetIDs (Chen, Lerman, Ferrara 2020; github.com/echen102/COVID-19-TweetIDs). For each monthly folder from January 2020 to February 2023, two hourly files (days 8 and 22 at 12:00 UTC; January 2020 used days 22 and 28 because collection began in late January, and February 2023 used days 8 and 15 because the day-22 file does not exist), 25 IDs drawn at random from each file with a fixed seed: 1,900 IDs in total, all with a decoded month matching their folder.
Method. One /twitter/tweets request per month (50 IDs each), 38 requests, plus one 51-ID request to confirm the limit (HTTP 400). One request timed out and was re-sent. An ID counts as returned if a tweet object with that id came back. Tweet year comes from the snowflake ID. Collected in 2026.
What the sample cannot show. Why a given ID is missing (the endpoint does not say), how other topics or collection methods behave, or how survival will change in future years. Monthly figures rest on 50 IDs each.
Privacy. The authors are private individuals, so raw responses stayed on the collection machine and were not published. The study files hold only aggregates and de-identified rows (year, month, returned or not, retweet flag, language), with no tweet IDs, URLs, text or handles.
# Hydrate a tweet ID file with twitterapi.io: 50 IDs per request, match by id, log the gaps.
import json, os, time
import requests
BASE = "https://api.twitterapi.io"
HEADERS = {"X-API-Key": os.environ["TWITTERAPI_IO_KEY"]}
ids = list(dict.fromkeys(l.strip() for l in open("ids.txt") if l.strip().isdigit()))
returned = missing = 0
with open("tweets.jsonl", "w") as out, open("missing.txt", "w") as miss:
for k in range(0, len(ids), 50):
batch = ids[k:k + 50]
r = requests.get(f"{BASE}/twitter/tweets",
params={"tweet_ids": ",".join(batch)},
headers=HEADERS, timeout=60)
r.raise_for_status()
by_id = {str(t["id"]): t for t in (r.json().get("tweets") or []) if t.get("id")}
for i in batch:
t = by_id.get(i)
if t is None:
miss.write(i + "\n"); missing += 1
continue
out.write(json.dumps({
"id": t["id"],
"createdAt": t["createdAt"],
"lang": t.get("lang"),
"text": t["text"],
"author": t["author"]["userName"],
"likeCount": t.get("likeCount"),
"retweetCount": t.get("retweetCount"),
"is_retweet": bool(t.get("retweeted_tweet")),
}, ensure_ascii=False) + "\n")
returned += 1
time.sleep(5) # 0.2 req/s before the first payment; lower this once you have paid
print(f"returned {returned}, missing {missing}, survival {returned / max(1, len(ids)):.1%}")Researching this?Free starter credits cover end-to-end testing — Google sign-in, no card. Get a key.
Questions readers ask
What does it mean to hydrate tweets?
Hydrating means taking tweet IDs and requesting the full tweet objects for them from an API, so you get the text, author, timestamp, language, engagement counts and media back. The reverse, keeping only the IDs, is called dehydrating, and it is how most research datasets are shared because X's developer policy allows redistributing Post IDs but restricts sharing the posts themselves.
Does twarc hydrate still work?
The twarc2 hydrate command still exists and needs a bearer token from an X developer account. twarc's README states that it is no longer actively supported after changes to Twitter's API quotas made it unusable, and X's API is now paid, so any run today is billed on X's rate card ($0.005 per post read on pay-per-usage). We did not test twarc for this page.
Is the DocNow Hydrator app still usable?
Not as shipped. Hydrator's README says the application keys it used for seven years have been rescinded by Twitter, and the app depends on those keys. Use a script against an API instead, such as the batch example on this page.
How many tweet IDs can I hydrate per request?
X's GET /2/tweets accepts up to 100 IDs per request. twitterapi.io's /twitter/tweets accepts up to 50 comma-separated IDs; we confirmed that 51 IDs returns HTTP 400.
How much does it cost to hydrate 100,000 tweet IDs?
On twitterapi.io the price is per tweet returned: $0.15 per 1,000 tweets, so $15 if every ID still exists and about $8.09 at the 53.9% survival rate we measured on a public 2020 to 2023 dataset. On X's pay-per-usage rate card, post reads are $0.005 each, so 100,000 posts that come back cost $500.
Why are so many tweet IDs missing when I hydrate?
Tweets that were deleted, tweets from accounts that went private, and tweets from suspended accounts are not returned. Retweets also disappear when the original post or the retweeting account is gone. In our sample of 1,900 IDs, 46.1% did not come back, and the share was higher for older tweets: only 45.5% of 2020 IDs still hydrated.
Can I tell whether a missing tweet was deleted or suspended?
With X's GET /2/tweets you get an errors entry per missing ID with a problem type such as resource-not-found or not-authorized-for-resource. twitterapi.io's /twitter/tweets simply omits the ID, and its docs say the reason can be deletion, a private account or a suspended account, so you can count returned versus not returned but not split the reasons.
Can I share the hydrated tweets with other researchers?
X's developer policy restricts redistributing X content and allows sharing Post IDs. It also lets academic researchers share IDs for non-commercial research and peer review, and permits up to 500 public post objects per person per day by non-automated means. Share the ID list plus your hydration counts, and let others hydrate it themselves.
Continue
- X Developer Policy: Content redistribution (Post IDs, 1,500,000 cap, academic exception)
- X API pricing: $0.005 per post read, pay-per-usage caps
- X API reference: GET /2/tweets, up to 100 IDs
- twarc on GitHub (README support status)
- COVID-19-TweetIDs dataset (Chen, Lerman, Ferrara 2020)
- twitterapi.io: get tweets by IDs (50 IDs per request)
- Twitter (X) API hub: endpoints, pricing and guides
- Building and exporting a Twitter dataset for research
- Twitter API for academic research after the free tier ended
- Where researchers get Twitter (X) data today
- Finding a deleted tweet: what still works
- Tweet ID to timestamp converter (snowflake decoder)
- twitterapi.io pricing and plans
- twitterapi.io API reference, including get tweets by IDs
- 01Most Mentioned Stocks and Crypto on Twitter (X): 81.3 Million Cashtag Mentions, 2024-2026
- 02Twitter (X) Images Not Loading: A Five-Hop Fix Checklist
- 03X Premium in 2026: The X Subscription Tiers, What They Unlock, and How to Cancel
- 04Twitter (X) Ads Library: What the X Ads Repository Shows, and What It Misses
- 05TweetScout explained: the crypto Twitter (X) score, its API, and a smart-follower score you can build yourself
- 06Mass Delete Tweets — Bulk API Workflow + Free Tool Options
- 07How to Download Tweets via API — A Python Guide
- 08How to Search Tweets by Date via API — A Developer's Guide
Writing about something you built with TwitterAPI.io? Our community rewards program pays API credits for honest tutorials, build logs and videos.
Stop reading. Start building.
Starter credits cover real testing on real data. Google sign-in, no card, no application queue.
Get an API keyAlready building with TwitterAPI.io? Publish an honest tutorial or build log about it and earn up to $200 a month in API credits.