snscrape for Twitter (X): why it returns blocked (404) and how to migrate

snscrape describes itself in its README as "a scraper for social networking services (SNS). It scrapes things like user profiles, hashtags, or searches and returns the discovered items, e.g. the relevant posts." It is a Python 3.8+ package by JustAnotherArchivist, GPL-3.0, with 5.5k stars, 785 forks and 466 commits on GitHub, and it covered eight services: Facebook, Instagram, Mastodon, Reddit, Telegram, Twitter, VKontakte and Weibo. The Twitter module was the one most people used, and it was broad: "users, user profiles, hashtags, searches (live tweets, top tweets, and users), tweets (single or surrounding thread), list posts, communities, and trends."

The interface had two halves. The CLI: snscrape twitter-user textfiles to get a user's posts, snscrape --max-results 100 twitter-hashtag archiveteam for a hashtag, and snscrape --jsonl [SCRAPER-NAME] [ARGUMENTS] to get every extracted field as one JSON object per line (those three are the README's own examples). And the module, which the README says "is currently undocumented" but which every tutorial used anyway: import snscrape.modules.twitter as sntwitter, then for tweet in sntwitter.TwitterSearchScraper("query since:2022-01-01 until:2022-02-01").get_items(): with a counter to stop at N, appending tweet.date, tweet.id, tweet.rawContent, tweet.user.username, tweet.likeCount and friends to a list that became a DataFrame. The twitter.py source defines ten scraper classes with CLI names: twitter-search, twitter-user, twitter-profile, twitter-hashtag, twitter-cashtag, twitter-tweet, twitter-list-posts, twitter-community, twitter-trends and twitter-users.

What made it popular is also what killed it. snscrape needed no API key and no account because it asked X's own web front end for the same JSON the browser loaded for a logged-out visitor. That was free, fast, had no documented rate limit and returned more fields than the official v1.1 API did. It also meant the data source was X's to remove, and in mid-2023 X removed it. The install still happens because the tutorials still rank; the error on the first run is the next section.

snscrape's Twitter module never authenticated. Before the first real request it called _ensure_guest_token(), which fetched a guest token from https://api.twitter.com/1.1/guest/activate.json (the same call the x.com front end made for anonymous visitors) and then sent that token with each request to the logged-out search, timeline and profile endpoints. The error strings in the module are the two ways that flow can fail: Unable to retrieve guest token when the activation call is refused, and blocked (404) when a data endpoint answers with a 404 for a request that used to work.

June 30, 2023 is the date. That day X required a login to view any post or search, and every snscrape Twitter scraper started returning blocked (404) four times (the module retries four times, then gives up) followed by a ScraperException. The maintainer opened issue #996, "All Twitter scrapes are failing: blocked (404)", the same day and pinned it, with the diagnosis: "This is likely connected to Twitter as a whole getting locked behind a login wall since earlier today. There is no known workaround at this time, and it's not known whether this will be fixable." X lifted the login requirement for viewing a single post a few days later, but the logged-out search and timeline JSON that snscrape read never came back. The thread now has over 160 comments, is still open, and comments posted in 2026 report the identical error.

Logged-in scraping was ruled out two years earlier. Issue #270, "Support for authentication", was opened by the maintainer in August 2021 and closed as wontfix with the reasoning that "snscrape has always been about making it easier to programmatically access _public_ social network service content", that accounts used for scraping would be banned, that testing with accounts was unreliable for the same reason, and that handling credentials safely would need substantial changes. So the one route that could have kept the module alive was out of scope by design, and the project did not change its mind after #996.

The duplicates tell you how often people still hit this: #997 "Yesterday i can, but today 404" (July 2023), #1003 "i am getting 404 block", #1013 "Does the Scrapper even work after Elon's API updates?", #1018 "ScraperException Error", #1048 "CRITICAL:snscrape.base:Errors: blocked (404), blocked (404)..." (July 2024), #1052 (December 2024) and #1056 "New 443 SSL Error" (February 2025), each closed as a duplicate of #996. A second open bug from the same day, #991, is the KeyError: 'timeline' crash on the twitter-profile scraper, which is the same login wall seen from a different endpoint.

So the diagnosis for a snscrape not working search does not depend on your setup: it is not your Python version, not a proxy, not the development branch (pip install git+https://github.com/JustAnotherArchivist/snscrape.git has the same module), and not a --retries or -vv flag. The PyPI release is 0.7.0.20230622 from June 22, 2023, eight days before the wall; the code asks X for data X no longer serves without a session. The rest of this page is what to do about it.

Be clear about the trade before writing code. snscrape was free, keyless and offline-installable, and it covered eight networks with one interface. A hosted Twitter (X) API gives up all three of those: you need a key, every tweet returned is billed, and the API covers X only. If your snscrape pipeline also pulled Reddit, Telegram or Mastodon, those modules are a separate question (the Reddit module uses Pushshift, which has its own history); this page only replaces the Twitter half.

The cookie libraries are the closest in spirit to snscrape: they act as logged-in accounts, so the account risk snscrape deliberately avoided is now yours, and the parser-chasing-the-front-end maintenance that killed the Twitter module is still there, in a library that is currently kept up. For a one-off academic pull that can be fine; for anything scheduled, the cost section's arithmetic usually says to pay for the reads instead.

The rest of this guide assumes the hosted-API route, because it is the one that maps onto snscrape's scrapers and fields without you operating accounts. The official X API is the other no-maintenance option and is in every cost table so you can see what it charges for the same reads. Python users coming from Tweepy rather than snscrape have their own Tweepy migration guide.

Each row is one scraper from snscrape/modules/twitter.py with its CLI name, and the request that replaces it. Endpoint paths are relative to https://api.twitterapi.io, all are GET with the X-API-Key header, and the response shapes are the ones in docs.twitterapi.io: advanced_search returns a top-level tweets array with has_next_page and next_cursor; last_tweets nests the list at data.tweets with the same two pagination fields at the top level and wraps errors as status: "error" with the reason in msg; user/info returns the profile in data.

Three details to carry into the code. First, advanced_search pages are "up to 20" tweets because ads and non-tweet cards are filtered out, so count rows, not pages, toward --max-results (in our study every page did hold exactly 20). Second, X search operators work inside query only on advanced_search; last_tweets takes no operators, which is why a dated pull on one account becomes from:handle since: until: on search. Third, a from: search does not return the account's retweets while last_tweets does; our search sample had 0 retweets in 800 tweets and the timelines had a few, so pick the endpoint by whether reposts matter to you. The search operators reference and the search-by-date guide cover the operator grammar in full.

This is the table that decides whether your downstream code changes. The left column is the Tweet and User dataclass in snscrape/modules/twitter.py (field names verbatim); the middle is the path in the tweet object that advanced_search and last_tweets return; the right is the share of tweets in our 2026 sample where that path was populated (search / timelines), so you know which columns will be full and which are only filled when the tweet has that thing.

Two readings of this table. The 18 core columns a typical snscrape DataFrame used (date, id, rawContent, user.username, the four counts, conversationId, lang) were populated on 100% of all 1,400 tweets in both collections, so a --jsonl-consuming pipeline needs renames, not null handling, for those. The conditional columns are where the two collections differ: links were on 40% of search tweets but 77% of organization timeline posts, mentions 10% versus 36%, and reply fields 16% versus 0% (the default includeReplies=false). Those are properties of what you query, not of the API.

We ran the two most common snscrape jobs through their replacements and measured what came back. The twitter-search stand-in was advanced_search with query=python pandas and queryType=Latest, followed for 40 pages; the twitter-user stand-in was last_tweets for @github, @Docker and @ThePSF, 10 pages each with the default includeReplies=false. All 70 data requests went through a client capped at 5 requests per second.

Pages are full. Every one of the 40 search pages and all 30 timeline pages held exactly 20 tweets, so --max-results 500 is 25 requests and the cost math is simply tweets x 15 credits. The docs say "up to 20" because ads and non-tweet cards are filtered out; on these queries nothing was filtered. The last search page still reported has_next_page: true, so a --max-results-less snscrape loop would have kept going; the wrapper stops on your limit, an empty page, or has_next_page false.

Search is not a timeline. The search sample was 77% original posts, 16% replies and 7% quotes, with 0 retweets in 800 tweets, from 413 distinct authors over 50 days. The three organization timelines were 95% originals, 4% quotes and 1% retweets, with no replies because includeReplies was off. snscrape's twitter-user included replies, so pass includeReplies=true if you want its behavior; snscrape's twitter-search excluded retweets too, so that half matches.

Text is long now. snscrape-era code often assumed 280 characters. In the search sample the median text was 292 characters, the 90th percentile 1,503 and the longest 22,480 (an article-length post); 56% of search tweets and 50% of timeline posts exceeded 280. A VARCHAR(280) column or a fixed-width CSV reader will truncate; text holds the whole post, there is no second field to stitch.

Pace. The 71 requests we timed took 170 seconds, 2.4 s per page, through a client that spaces requests at least 0.2 s apart and was shared with other jobs on the same machine at the time; treat it as what a careful script sees, not as an API benchmark. New accounts are limited to 0.2 requests per second until the first payment (20 per second after), so on a free account the same 50 pages take about four minutes.

One account came back empty, and the profile call explained why. Our third planned timeline, @python_org, returned status: "success" with an empty data.tweets and has_next_page: false. GET /twitter/user/info?userName=python_org returned HTTP 200 with data.unavailable: true and unavailableReason: "Suspended" rather than an error status, so check data.unavailable yourself (the wrapper's twitter_profile() does). We replaced the account with @ThePSF (692,722 followers, 6,954 posts per user/info).

snscrape cost nothing in money and, after June 2023, returned nothing. The replacement is metered, so here is the meter. Prices are from the twitterapi.io pricing page and the official X API pay-per-usage page; the ratios are arithmetic, not a benchmark. The study on this page is the first row: 1,400 tweets plus 2 profile lookups and 1 empty page cost $0.21; the same reads at the official rates would be $7.02.

The worked example most snscrape users care about is a scheduled job: 10,000 tweets a day is 10,000 x 15 = 150,000 credits = $1.50 a day at $1 per 100,000 credits, or $45 over 30 days; the official route for the same reads is 10,000 x $0.005 = $50 a day, $1,500 a month. New accounts that pass the automatic bot check get $1 of free credits (100,000 credits, no card; one per person or network; the credits expire after 31 days), which is about 6,600 tweets, enough to rerun your old snscrape query through the wrapper and diff the two JSON Lines files before paying anything. Subscriptions start at Starter $29 a month; the pricing page has the tiers and bonus credits.

One billing rule changes how you write the loop: you pay per tweet returned, so a narrow query is cheap and an over-broad one is not. A since:-bounded daily pull on a quiet topic costs the 15-credit minimum ($0.00015) on days it finds nothing and $0.003 per full page otherwise. snscrape users who used to pull everything and filter in pandas should move the filter into the query (lang:en, min_faves:, -filter:replies) where it reduces the bill.

The main code block at the end of this page is snscrape_compat.py. It re-implements the four snscrape entry points most pipelines used, with the same CLI shape (--jsonl, --max-results, scraper name, argument) and classes with the same names and a get_items() generator, so both the shell one-liners and the module loops in old notebooks change by one import. Each item is a dict with snscrape's field names, built by to_snscrape() from the mapping table above; date is ISO 8601 in UTC as snscrape's --jsonl printed it, user is a nested object, and quotedTweet / retweetedTweet recurse. The pandas block shows that pd.read_json(lines=True) on the output reads the same way as before.

The wrapper does not emulate renderedContent, vibe, editState or conversationControlPolicy (nothing to map them to; code that read those columns gets a KeyError rather than a silently empty column), nor twitter-trends, twitter-community or twitter-list-posts, which are one-line additions following the table above.

How we collected this. Question: what does a migrated snscrape job return today, column by column, page by page? Query: python pandas, queryType=Latest, 40 pages of /twitter/tweet/advanced_search; accounts: @github, @Docker and @ThePSF, 10 pages each of /twitter/user/last_tweets with the default includeReplies=false; @python_org was planned, returned an empty timeline, and was replaced after /twitter/user/info reported it unavailable. Window: the most recent posts at collection time (the search sample spanned 50 days, the timelines 118 to 353 days). Sample: 1,400 tweets, 73 requests in total. Collected in 2026 through a read-only client capped at 5 requests per second. A field counts as populated when it is present and not null, not an empty string, list or object; counts of 0 are populated.

What the sample cannot show. One developer-topic query and three organization accounts are not all of X. Reply, media and link shares depend heavily on the query and the account type; organization accounts post more links and fewer replies than individuals, and a consumer topic would have a different language mix than the 77% English we saw. The pace figure reflects a shared rate limiter, not the API's ceiling. We did not test twitter-list-posts, twitter-community or twitter-trends; their rows in the table come from the endpoint reference, not from this run.

Aggregates only. Posts and accounts named on this page are the organization accounts listed above; search authors are not identified, and the raw search pages were not kept beyond the aggregates. The analysis script, results and the gzip-compressed raw timeline pages for the organization accounts are kept with the study.