Social Media Scraper: Choosing a Collector per Network, with Twitter (X) via API
- No universal scraper. Each network has its own access route, limits and terms, so plan one collector per network and merge rows into one schema.
- Brands rarely cross-link. Only 1.7% of 757 original brand posts on X linked to another social network (collected in 2026), so crawling one network will not find the others.
- X via API. Our 50-request pull returned 1,000 tweets for $0.15 ($0.15 per 1,000), versus $5.00 per 1,000 post reads on X's official pay-per-usage rate card.
- YouTube and Bluesky have open APIs. YouTube Data API: 10,000 default quota units a day. Bluesky:
public.api.bsky.appanswered our profile request with no login. - Instagram and TikTok are gated. Meta's APIs center on professional accounts; TikTok's Research API is for eligible researchers, not commercial buyers.
- Rich X records. 100.0% of sampled posts carried a
viewCount, 59.8% carried media, and 31.0% of original posts were video.
People searching for a social media scraper usually want one thing: a single tool that pulls posts, profiles and engagement from every network into one spreadsheet. The market sells that promise in two forms, multi-network scraping platforms and multi-network data APIs, and both hide the same fact underneath: every network is collected differently, priced differently and governed by different terms.
This page answers the multi-network buyer question: what to use per network, and how to get X (Twitter) data through an API instead of a page scraper. If you only need X and want a vendor-by-vendor comparison, read our Twitter (X) scraper comparison instead; this page does not repeat its table. If you are still deciding between scraping and APIs in general, the web scraping vs API guide covers that choice. We are honest about scope: TwitterAPI.io covers X only.
Everything numeric here comes from a study we ran ourselves: 1,000 posts from ten brand accounts on X, collected in 2026, with the request count and the bill.
1.7% of 757 original posts
What a social media scraper actually does
A social media scraper turns public social content into structured rows: who posted, when, the text, the media, and the counters (likes, reposts, replies, views). It does that in one of three ways, and the way matters more than the brand name on the tool.
Official API. The network itself sells or grants access: X's API, the YouTube Data API, Meta's Instagram Platform, TikTok's Research API, Reddit's Data API. You get documented fields and documented limits, and you accept the network's terms and price.
Third-party data API. A vendor runs the collection and hands you JSON over HTTP. You call an endpoint, you get rows, you pay for what comes back. TwitterAPI.io is this kind of product for X.
Page scraper. A headless browser or HTTP client loads the public web pages and parses the HTML. It is flexible, but it breaks when the page layout changes, often needs logged-in sessions and proxies, and leaves you responsible for the network's terms of service.
Most commercial social media scrapers are bundles: a platform hosting many small page scrapers, one per network, sold under one bill. That can be convenient, but the reliability and cost of each network still depend on how that one network is collected.
The per-network map: which route fits each network
Here is the access route we would pick for each major network, with the fact we verified this session from the network's own documentation.
Why the map matters. A multi-network scraping platform does not change these facts; it either uses these routes for you or scrapes pages where no route fits your use. Ask any vendor which of the two it uses per network, because that answer predicts how often your pipeline breaks.
The gated networks. For Instagram and TikTok, the official routes do not serve a general commercial buyer who wants arbitrary public accounts. That is exactly where page scrapers are sold, and exactly where terms of service and account restrictions bite hardest. Read the network's terms before you buy.
Our study: brand posts on X rarely point at their other networks
A common shortcut is to scrape one network and follow links to the others. We tested that on X. We pulled the latest 100 posts from each of ten brand and media accounts and classified every outbound link in the account's own (non-repost) posts by destination.
Pooled across 757 original posts, 41.0% linked to any outside site, but only 1.7% linked to another social network (12 YouTube links and 1 Reddit link, nothing to Instagram, TikTok, LinkedIn, Facebook, Threads or Bluesky). News accounts link to their own articles; entertainment and sports accounts post native video and rarely link at all.
Takeaway. You cannot discover a brand's Instagram or TikTok output by crawling its X feed. Treat each network as its own source, keyed by the brand's handle on that network, and keep a mapping table of handles per brand.
X (Twitter): why an API beats a page scraper
X is the network where buyers most often reach for a page scraper, and where it is least needed. Since the June 30, 2023 login wall, most of x.com requires a logged-in session to view, so a page scraper has to drive real accounts, and those accounts carry the risk of rate limits, locks and suspension. A data API removes that from your side: no X account needed for reads.
What one record carries. Each tweet from /twitter/user/last_tweets or /twitter/tweet/advanced_search arrives as JSON with id, text, createdAt, lang, likeCount, retweetCount, replyCount, quoteCount, viewCount, bookmarkCount, entities, extendedEntities.media, quoted_tweet, retweeted_tweet and an author object. In our sample, 100.0% of original posts had a non-zero viewCount, 100.0% had lang, 59.8% had media and 28.8% had a link card.
What brands post. Of the original posts, 31.0% carried video, 28.8% photos and 40.2% no media. If you need the media files themselves, the URLs are in extendedEntities.media[].media_url_https and video_info.variants[]; a page scraper would have to rebuild all of that from the HTML.
The endpoints a social media pipeline uses on X. /twitter/user/last_tweets for an account's timeline (20 per page, newest first), /twitter/tweet/advanced_search for keyword and operator queries, /twitter/user/mentions for brand mentions, /twitter/user/info for profiles and /twitter/user/followers for audiences. Each returns has_next_page and next_cursor for paging.
What the X part costs, measured
We billed our own study, so the number is real rather than a rate card. 50 requests returned 1,000 tweets (20 tweets per page). At 15 credits per tweet returned, that is 15,000 credits, or $0.15: $0.15 per 1,000 tweets.
That is about 33x apart per 1,000 tweets ($5 against $0.15). X's pay-per-usage plan is capped at 3 million post reads a month, with Enterprise above that; TwitterAPI.io has no monthly cap and you pay only for the data returned. Profiles cost 18 credits each ($0.18 per 1,000), and follower or following lists cost 1 to 3 credits per user returned.
The empty-result floor. A request that returns nothing still bills the 15-credit minimum ($0.00015). That matters for monitoring: checking a quiet account every 60 seconds for a month is 43,200 checks, or $6.48 if every check comes back empty; every 5 minutes is $1.30.
New accounts that pass the automatic bot check get up to $1 of free credits (at most 100,000 credits, no card, one per person or network, expiring after 31 days). Purchased credits never expire. Rate limits are 0.2 requests per second before the first payment and 20 once paid. Plans and subscriptions are on the pricing page.
YouTube, Instagram, TikTok, Bluesky and Reddit: what to use
YouTube. The YouTube Data API v3 is the route. Google's docs give each project a default of 10,000 quota units a day for most endpoints, with list reads such as videos.list and commentThreads.list costing 1 unit each, plus a separate daily allocation of 100 search.list calls. Search is the scarce resource: resolve channels once, then read their uploads playlists.
Instagram. Meta's Instagram Platform is built around Instagram Business and Creator accounts: the Instagram API with Instagram Login and with Facebook Login both serve professional accounts, and hashtag search is listed under the Facebook Login variant. It fits a brand analyzing its own account, less so a buyer collecting arbitrary public accounts.
TikTok. TikTok's Research API is open to academic institutions in the U.S., EEA, UK, Canada and Switzerland, to not-for-profit research bodies in the EU and to some others under listed conditions, all of whom must be independent of commercial interests. A commercial buyer will not qualify; plan for that before you promise TikTok data.
Bluesky. Bluesky is the open case. We called public.api.bsky.app/xrpc/app.bsky.actor.getProfile for the bsky.app account with no login and got HTTP 200 and a full profile. Our Bluesky API guide covers its auth, limits and feeds.
Threads and Reddit. Threads has its own Meta API (see our Threads API guide). Reddit's Data API Wiki on support.reddithelp.com could not be fetched this session, so we state no Reddit limits here: read Reddit's current terms before building on it.
One schema for many networks
The hard part of multi-network collection is not the fetch, it is making rows comparable. Normalize every network into one table with a network column and keep the native fields in a JSON column for later.
Do not compare raw counts across networks. A view on X, a view on YouTube and a play on TikTok are counted differently. Compare within a network, or use engagement rates computed the same way per network.
Deduplicate reposts. In our sample, reposts ranged from 7% of @NBA's latest 100 posts to 68% of @Microsoft's. Decide whether a repost counts as the brand's content before you aggregate, and filter on retweeted_tweet accordingly.
def normalize_x(t):
"""Map one TwitterAPI.io tweet to the shared social-post schema."""
return {
"network": "x",
"post_id": str(t["id"]),
"created_at": t["createdAt"],
"text": t.get("text", ""),
"likes": t.get("likeCount", 0),
"shares": t.get("retweetCount", 0) + t.get("quoteCount", 0),
"comments": t.get("replyCount", 0),
"views": t.get("viewCount", 0),
"author": (t.get("author") or {}).get("userName"),
"is_repost": bool(t.get("retweeted_tweet")),
}
How to choose a social media scraper: a buyer checklist
1. List networks by priority. Most projects have one network that matters most. Choose that route first and accept a weaker route for the rest.
2. Ask how each network is collected. Official API, data API or page scraper, per network. Page scraping of logged-in networks carries account risk and breaks on layout changes.
3. Price per result, not per run. Run a small paid pull, count the rows that came back, divide. Our X pull cost $0.15 per 1,000 tweets because we billed only for the 1,000 tweets returned.
4. Check the fields you need. Views, media URLs, quote and repost structure and author follower counts are not universal. Pull 100 rows and inspect them before you commit.
5. Read the terms. Each network's developer terms decide what you may store, how long and for what purpose. Personal data of private individuals carries extra duties under privacy law.
6. Plan for monitoring cost. Polling multiplies requests. On X, empty checks bill a $0.00015 minimum each, so set a sensible interval or use push delivery.
How we collected this
Question. How much of a brand's X feed points at its other social networks, what a structured X record carries, and what a real pull of X data costs per 1,000 tweets.
Accounts. Ten public organization accounts: @NASA, @Spotify, @netflix, @nytimes, @espn, @YouTube, @NBA, @Microsoft, @NatGeo, @BBCWorld.
Method and window. /twitter/user/last_tweets with default settings (replies excluded), 5 pages per account, newest first; each account's window runs from its oldest to newest sampled post, from 1.7 days for @NBA to 160.8 days for @Microsoft. Collected in 2026. Sample: 1,000 posts, 757 original (non-repost) posts, 50 requests, $0.15 billed.
Link classification. Each entities.urls[].expanded_url host was mapped to a network list (YouTube, Instagram, TikTok, LinkedIn, Facebook, Threads, Bluesky, Reddit, Twitch, Spotify); links back to X itself were ignored, everything else counted as outside web. Shortened links that were not expanded would count as outside web, so the social-link share is a floor.
What the sample cannot show. Ten accounts are not all brands; a different set (consumer brands, creators) may cross-link more. Brands may also link their networks from their profile bio rather than from posts, which this study does not count. Views are X's own counter at collection time.
Privacy. Aggregates only, organization accounts only; no reposted, quoted or mentioned authors were analyzed. De-identified rows live in data/studies/social-media-scraper/sample.json.
import os
import time
from collections import Counter
from urllib.parse import urlparse
import requests
BASE = "https://api.twitterapi.io"
HEADERS = {"X-API-Key": os.environ["TWITTERAPI_IO_KEY"]}
ACCOUNTS = ["NASA", "NatGeo", "YouTube"]
SOCIAL = {
"youtube.com": "youtube", "youtu.be": "youtube", "instagram.com": "instagram",
"tiktok.com": "tiktok", "linkedin.com": "linkedin", "facebook.com": "facebook",
"threads.net": "threads", "threads.com": "threads", "bsky.app": "bluesky", "reddit.com": "reddit",
}
def network(url):
host = (urlparse(url).hostname or "").lower().removeprefix("www.")
for dom, name in SOCIAL.items():
if host == dom or host.endswith("." + dom):
return name
return "web"
def last_tweets(user, pages=5):
"""Newest-first timeline, 20 tweets per page, billed per tweet returned."""
cursor, out = "", []
for _ in range(pages):
r = requests.get(
f"{BASE}/twitter/user/last_tweets",
headers=HEADERS,
params={"userName": user, "cursor": cursor},
timeout=30,
)
r.raise_for_status()
body = r.json()
if body.get("status") == "error":
raise RuntimeError(body.get("msg"))
data = body.get("data") or {}
out.extend(data.get("tweets") or [])
if not body.get("has_next_page"):
break
cursor = body.get("next_cursor") or ""
time.sleep(0.3)
return out
rows = []
for acct in ACCOUNTS:
for t in last_tweets(acct):
if t.get("retweeted_tweet"):
continue # keep the account's own posts only
urls = [u.get("expanded_url", "") for u in (t.get("entities") or {}).get("urls", [])]
media = (t.get("extendedEntities") or {}).get("media") or []
rows.append({
"network": "x",
"account": acct,
"id": t["id"],
"created_at": t["createdAt"],
"text": t["text"],
"likes": t.get("likeCount", 0),
"reposts": t.get("retweetCount", 0),
"replies": t.get("replyCount", 0),
"views": t.get("viewCount", 0),
"media_type": media[0]["type"] if media else "none",
"links_to": sorted({network(u) for u in urls if u}),
})
print(len(rows), "original posts")
print(Counter(n for r in rows for n in r["links_to"]).most_common())
Researching this?Free starter credits cover end-to-end testing — Google sign-in, no card. Get a key.
Questions readers ask
What is a social media scraper?
Software that collects public posts, profiles and engagement counts from social networks into structured data. It works through an official API, a third-party data API or by parsing web pages, and the method decides its reliability, cost and terms.
Is there one scraper that covers every social network?
Multi-network platforms exist, but underneath they still collect each network differently. Ask how each network is collected, because a network collected by page scraping breaks and costs differently from one collected through an API.
Is scraping social media legal?
It depends on the data, the network's terms and your jurisdiction. Public data is not automatically free to reuse, personal data of private individuals carries privacy-law duties, and each network's developer terms decide what you may store. Take legal advice for your use case.
What is the best way to get X (Twitter) data?
Through an API rather than a page scraper, because most of x.com requires a login. In our study 1,000 tweets cost $0.15 on TwitterAPI.io ($0.15 per 1,000), against $5.00 per 1,000 post reads on X's official pay-per-usage rate card. No X account is needed for reads.
Can I scrape Instagram and TikTok through an official API?
Only in limited cases. Meta's Instagram Platform APIs are built for Business and Creator accounts, and TikTok's Research API is open to eligible academic and not-for-profit researchers who are independent of commercial interests.
How do I get YouTube data without scraping?
Use the YouTube Data API v3. Google's docs give a default of 10,000 quota units a day for most endpoints, list reads cost 1 unit, and search has a separate daily allocation of 100 calls, so resolve channels once and read their uploads playlists.
Does TwitterAPI.io scrape Instagram, TikTok or YouTube?
No. TwitterAPI.io covers X (Twitter) only: tweets, profiles, followers, search, mentions and real-time delivery. For other networks use the routes on this page.
How can I try it without paying?
New accounts that pass the automatic bot check get up to $1 of free credits (at most 100,000 credits, no card, one per person or network, expiring after 31 days). That covers a test pull of a few thousand tweets.
Continue
- X API pricing: pay-per-usage rates ($0.005 per post read) and the 3M post-read monthly cap
- YouTube Data API: quota costs and default daily allocation
- Meta Instagram Platform: APIs for Business and Creator accounts
- TikTok Research API: eligibility
- Bluesky developer docs: API directory and public AppView
- TwitterAPI.io: user last tweets endpoint (20 per page, billed per tweet returned)
- X (Twitter) API hub: endpoints, pricing and guides
- Twitter (X) scraper comparison: X-only tools side by side
- Web scraping vs API: when to use each
- Apify pricing explained for X data
- Bright Data alternatives for social data
- PhantomBuster alternatives for X automation
- Social media API: the four types explained
- Bluesky API: auth, limits and cost
- Threads API (Meta): limits compared with X
- TwitterAPI.io pay-per-use pricing
- 01Free LLM API options in 2026: what is still free, the real limits, and how to feed it Twitter (X) data
- 02RAG vs MCP: Retrieval vs Live Tools, Measured on Twitter (X) Data
- 03Clay Pricing in 2026: Plans, Data Credits, Actions and the Cost of a Twitter (X) Column
- 04MCP Inspector: How to Test and Debug an MCP Server, Tested on a Live X Data Server
- 05What Is an API Key? Meaning, a Real Request, and How to Keep One Safe
- 06Social Media API Explained: Four Kinds, and What the Twitter (X) Data Costs
- 07Sprout Social Alternatives: What to Use Instead, by Job, With a Twitter (X) Data Study
- 08Sprout Social API in 2026: What It Returns, What It Costs, and Where Twitter (X) Data Runs Out
Writing about something you built with TwitterAPI.io? Our community rewards program pays API credits for honest tutorials, build logs and videos.
Stop reading. Start building.
Starter credits cover real testing on real data. Google sign-in, no card, no application queue.
Get an API keyAlready building with TwitterAPI.io? Publish an honest tutorial or build log about it and earn up to $200 a month in API credits.