Twitter (X) Dataset Export — A Researcher's Guide
Academic and research use of Twitter (X) data has been reshaped since the sunset of the v2 Academic tier. In 2026 researchers building datasets have two practical paths: enterprise-tier X official (contract-required) or third-party APIs like twitterapi.io (email signup, no gating).
This guide walks the practical dataset-export pipeline — from query to JSONL archive to analysis-ready formats — with the honest cost math + the ethics considerations peer-review requires. Pricing references URL-cited.
Design your dataset before you query
Research question first: What are you actually trying to measure? Discourse trends over years? Sentiment during a specific event? Behavioral patterns of specific accounts? The question shapes the query.
Query specification: since: / until: bounds the time window. from:user or keyword filters bound the topic. min_faves: / lang: narrow signal. Write out the exact operator expression before you start querying — it becomes part of your methods section.
Sample size + statistical power: If you're doing statistical analysis, estimate needed sample size upfront. If discourse analysis, 500-5000 tweets is typical; sentiment on a moderate topic, 10-50K; longitudinal / cross-year, 100K-1M+.
Ethics + IRB pre-registration: Register your protocol with your institution's IRB or ethics board before data collection. Some journals require pre-registration for observational social-media research.
The export pipeline — query to archive to analysis
Step 1 — Query: cursor-paginate /twitter/tweet/advanced_search until exhausted or budget cap hit. Write each result to JSONL for archival.
Step 2 — Archive: JSONL is the canonical format for research archival. Append-only, streamable, minimal parsing. One JSON per line. Store with captured_at metadata per row.
Step 3 — Analysis format: derive CSV or Postgres from the JSONL for actual analysis. CSV for one-off statistical work in R / Python; Postgres for query-heavy multi-dimensional analysis.
Step 4 — Reproducibility artifacts: store the exact query string + count + captured_at range alongside the archive. Reviewers can rerun the query to verify.
Runnable pipeline
Auth via X-API-Key. Pricing per twitterapi.io/pricing: $0.00015 per returned tweet.
import os, requests, json, csv
from pathlib import Path
from datetime import datetime, timezone
HEADERS = {"X-API-Key": os.environ["TWITTERAPI_IO_KEY"]}
BASE = "https://api.twitterapi.io"
def export_dataset(query: str, out_dir: str = "corpus"):
"""Full-archive query → JSONL archive + CSV + query-metadata sidecar."""
out = Path(out_dir)
out.mkdir(exist_ok=True)
started_at = datetime.now(timezone.utc).isoformat()
tweets, cursor = 0, None
jsonl_path = out / "corpus.jsonl"
with open(jsonl_path, "w") as f:
for _ in range(1000): # generous cap for research pulls
params = {"query": query}
if cursor: params["cursor"] = cursor
r = requests.get(f"{BASE}/twitter/tweet/advanced_search", headers=HEADERS, params=params, timeout=15)
r.raise_for_status()
resp = r.json()
for t in resp.get("tweets", []):
f.write(json.dumps(t) + "\n")
tweets += 1
cursor = resp.get("next_cursor")
if not cursor: break
# Reproducibility sidecar
sidecar = {
"query": query,
"tweets_returned": tweets,
"started_at": started_at,
"completed_at": datetime.now(timezone.utc).isoformat(),
"provider": "twitterapi.io",
"endpoint": "/twitter/tweet/advanced_search",
}
with open(out / "query_metadata.json", "w") as f:
json.dump(sidecar, f, indent=2)
# CSV derivation for statistical analysis
with open(jsonl_path) as jf, open(out / "corpus.csv", "w") as cf:
writer = csv.DictWriter(cf, fieldnames=["id", "created_at", "author", "text", "likes", "retweets", "lang"])
writer.writeheader()
for line in jf:
t = json.loads(line)
pm = t.get("public_metrics", {})
writer.writerow({
"id": t.get("id"),
"created_at": t.get("created_at"),
"author": t.get("author", {}).get("userName", ""),
"text": t.get("text", "").replace("\n", " "),
"likes": pm.get("like_count", 0),
"retweets": pm.get("retweet_count", 0),
"lang": t.get("lang", ""),
})
print(f"exported {tweets} tweets to {out}/ (JSONL + CSV + metadata sidecar)")
return tweets
# Example: climate change discourse 2020-2024
export_dataset('"climate change" since:2020-01-01 until:2025-01-01 lang:en min_faves:5')Cost estimation for common corpus sizes
Math from twitterapi.io/pricing at $0.00015 per returned tweet.
10K tweets (pilot / method development): $1.50
100K tweets (moderate discourse study): $15
500K tweets (longitudinal or multi-year): $75
1M tweets (large-scale cross-language or multi-topic): $150
10M tweets (platform-level statistical study): $1,500
Storage cost typically dwarfs API cost at these sizes. Budget for warehouse + archival storage separately.
Ethics + IRB considerations
Public data doesn't automatically = ethics-neutral: even public tweets belong to identifiable individuals. IRB / ethics review applies.
Identifiability handling: for aggregate analysis, tweets are usually reportable as counts / distributions without identifying individuals. For qualitative analysis (quoting specific tweets), consider whether to anonymize handles per your ethics guidance.
Vulnerable populations: research involving tweets from minors, medical patients, refugees, other vulnerable groups requires stricter ethics protocols even for public data.
Data retention: some ethics boards require deletion of raw data after study completion. Others require preservation for verification. Follow your protocol.
Third-party API compliance: check your ethics board's stance on third-party data providers. Some prefer X official; twitterapi.io is generally acceptable given the data is X's public data (not proprietary or scraped). Document the provider + endpoint in your methods.
Peer-review citation + reproducibility
Cite the provider: 'Data collected via twitterapi.io's /twitter/tweet/advanced_search endpoint on
Store the raw JSONL: reviewers may request. Institutional archival is standard.
Store the query metadata: alongside the corpus. Query + count + captured_at makes reruns possible.
Note deletion possibility: tweets deleted between your capture + review may not be reproducible. Note this limitation in methods.
Version + timestamp your analysis code: git tag the analysis scripts. Reviewers can verify by running against your archive.
Side-by-side — dataset export paths
twitterapi.io wins on setup friction + individual-researcher accessibility; X official Enterprise wins for institutional programs where the contract is already in place.
# Practical example: reproducible longitudinal corpus with per-year splits.
import os, requests, json
from pathlib import Path
from datetime import date, timedelta, datetime, timezone
HEADERS = {"X-API-Key": os.environ["TWITTERAPI_IO_KEY"]}
BASE = "https://api.twitterapi.io"
TOPIC = '"artificial intelligence" min_faves:10 lang:en'
YEARS = range(2020, 2026)
OUT_DIR = Path("ai_discourse_corpus")
OUT_DIR.mkdir(exist_ok=True)
total_tweets = 0
total_cost = 0.0
for year in YEARS:
year_start = date(year, 1, 1).isoformat()
year_end = date(year + 1, 1, 1).isoformat()
query = f"{TOPIC} since:{year_start} until:{year_end}"
tweets, cursor = 0, None
with open(OUT_DIR / f"{year}.jsonl", "w") as f:
for _ in range(500):
params = {"query": query}
if cursor: params["cursor"] = cursor
r = requests.get(f"{BASE}/twitter/tweet/advanced_search", headers=HEADERS, params=params, timeout=15)
r.raise_for_status()
resp = r.json()
for t in resp.get("tweets", []):
f.write(json.dumps(t) + "\n")
tweets += 1
cursor = resp.get("next_cursor")
if not cursor: break
year_cost = tweets * 0.00015
total_tweets += tweets
total_cost += year_cost
print(f" {year}: {tweets} tweets (${year_cost:.2f})")
print(f"\nTotal: {total_tweets} tweets, ${total_cost:.2f}")
# Reproducibility sidecar
sidecar = {
"topic": TOPIC,
"years": list(YEARS),
"total_tweets": total_tweets,
"total_cost_usd": round(total_cost, 2),
"provider": "twitterapi.io",
"endpoint": "/twitter/tweet/advanced_search",
"exported_at": datetime.now(timezone.utc).isoformat(),
}
with open(OUT_DIR / "corpus_metadata.json", "w") as f:
json.dump(sidecar, f, indent=2)
# Cost per twitterapi.io/pricing: linear per-tweet, no minimum
# Peer-review-quality reproducibility: query + count + timestamp per yearQuestions readers ask
Can I publish research using twitterapi.io data?
Yes — cite the provider, endpoint, and access date in your methods section. Peer-review generally accepts third-party API data sources when documented and reproducible. Check journal-specific data policies.
What formats work best for downstream analysis?
JSONL for archival (append-friendly, streamable). CSV for statistical software (R / Python pandas / SPSS). Postgres for query-heavy multi-dimensional analysis. All derivable from a single JSONL source.
How do I handle deleted tweets in my dataset?
Tweets deleted after your export remain in your archive. Note this in methods — your corpus reflects the state at capture time. Some analyses use deletion as a signal (e.g., studying content moderation); most treat the capture-time snapshot as the dataset.
Can I share my dataset publicly?
Check X's developer terms + your provider's terms — public tweet redistribution has limitations. Many researchers share the tweet IDs (which reviewers can re-fetch) rather than the full content, which respects deletion + platform terms.
What about non-English research?
Filter by lang:xx (ISO 639-1 codes). Multi-language corpora: run separate queries per language + combine at analysis time.
Rate limits during large export?
twitterapi.io per-key throughput handles thousands of requests/hour. For very large exports (millions of tweets), pace your pulls across hours rather than bursting to avoid hitting soft caps.
Continue
- twitterapi.io — pricing
- X API — pricing (docs.x.com, 2026 verified)
- X — full-archive search (Enterprise)
- Twitter (X) API — cluster hub
- Twitter (X) academic research API guide
- Twitter (X) history API — export timeline
- Twitter (X) data collection API developer guide
- Twitter (X) advanced search — complete guide
- twitterapi.io pricing
Stop reading. Start building.
Starter credits cover real testing on real data. Google sign-in, no card, no application queue.
Get an API key