R.I.P. Nitter 🪦😭 (...unless?) #919
Labels
No labels
accessibility
bug
discussion
documentation
enhancement
feature parity
good first issue
help wanted
question
rss
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
PAS/nitter#919
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
https://techcrunch.com/2023/06/30/twitter-now-requires-an-account-to-view-tweets/
the nitter crawler will need to be recreated...
it was working this morning but it appears nitter is now broken. viewing any profile results in the following error being thrown
Libreddit gone tomorrow.
Wonder how long Invidious and FreeTube will last?
It's so over
the twitter rank has already dropped 6 places from yesterday to today 🤦♂️ https://ciscorank.com/s/twitter.com
At least reddit still has the old.reddit.com. Which is a pretty descent front-end (as long as you don't care with privacy), with twitter, you are pretty much forced to use that horrible garbage interface. But yeah, this data rush pretty much will make all platforms to garden walled themselves.
There are plenty of people who only engage in the modern internet through front-ends; myself being one of them. I have no interest in registering for sites; it took me ages to register a proper git.
One wonders what will happen when these self-inflicted stab-wounds deplete 'profit' rather then generate it?..
Removing access isn't going to make people want to use a site more.
And it certainly won't inflate user numbers, which I'd imagine is the end goal.
C'est la vie.
I'm glad I got to experience the internet as it was.
thoughts?
No chance in hell this is temporary unless it affects ad revenue
Maybe people will be smart enough to stop using proprietary garbage finally...
"Temporary emergency" measure that will always be in place forever.
This fork is working
https://github.com/PrivacyDevel/nitter/
But privacydev.net isn't.
I can confirm after building that fork that it also has the same issue
That instance is always pegged from the sheer volume of traffic alone.
"There is nothing more permanent than a temporary measure". Either now or in the future.
https://github.com/miguelfreitas/twister-core
https://github.com/szelok/twister-android
It's time to admit: providing an alternative frontend to Twitter was an idea determined to fail. What should have been done, is mirroring the content of Twitter into Twister automatically, this way having more content in twister than in twitter and creating incentives for users to join twister instead of twitter.
Nitter.net returns the following "Jester route exception" which seems quite unusual for the twitter gateway not being accessible:
"An error has occured in one of your routes.
Detail: /home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) getGraphUserNimAsyncContinue
/home/ubuntu/nitter/src/api.nim(13) getGraphUserIter
/home/ubuntu/nitter/src/experimental/parser/graphql.nim(7) parseGraphUser
/home/ubuntu/.nimble/pkgs/jsony-#ea811be/jsony.nim(590) fromJson
/home/ubuntu/.nimble/pkgs/jsony-#ea811be/jsony.nim(45) parseHook
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) getCachedUserNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) getCachedUserIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) fetchProfileNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) fetchProfileIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) showTimelineNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) showTimelineIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) matchNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) matchIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) dispatchNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) dispatchIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
[[reraised from:
/home/ubuntu/nitter/src/nitter.nim(61) nitter
/home/ubuntu/.nimble/pkgs/jester-#baca3f/jester.nim(511) serve
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1956) runForever
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1653) poll
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(1394) runOnce
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncdispatch.nim(234) processPendingCallbacks
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(28) handleRequestSlowNimAsyncContinue
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncmacro.nim(131) handleRequestSlowIter
/home/ubuntu/.choosenim/toolchains/nim-1.6.10/lib/pure/asyncfutures.nim(389) read
]]
Expected { but end reached. At offset: 0"
Thanks anyway for providing and maintaining Nitter.
This is a sad year for social media.
Cheers lads, was a wild ride.
Couldn't we use accounts to scrape the data (one per instance)? As far as I know this was already done for NSFW-content so maybe it could be applied to all content now
There is one instance that is still functional:
https://tweet.whateveritworks.org
Partly functional, anyway...individual tweets are displayed, but profiles produce a 502 error.
Well, big thanks to @zedeus for nitter, have been enjoying it for some time.
Hope it can be fixed, but... it was going downhill for some time already.
Around a month ago twitter stopped working with disabled cookies, now this.
Okay, well, to Mastodon I guess. It's a shame they're killing this. I don't think there are many here who will respond to this by going back to regular Twitter. I'm now just done with Twitter for good I suppose. Thanks to everyone who tried their best.
Almost certainly yes, but 1) this requires registering burner accounts, possibly with some regularity as I suspect they'll get banned, and 2) the accounts will need to be linked to phone numbers because Twitter requires this if your account looks suspicious at all.
I don't think this is all that hard but it adds extra cost on top of running a server and it's annoying.
Thanks Nitter devs, you have been heroes of privacy, for a while. :)
I don't think this will affect me much, because I have not been interested in Twitter's content for a while. This network has been slowly and painfully dying since Musk bought it (and even before that, it became mostly a trolling platform for political people to hate each others), so whatever. Most of my social feeds go through Mastodon and Lemmy, nowadays, anyway. This is the end of an era, what comes next is exciting!
Glad to see all sorts of "social" media dying 😍
Can you scrape the embedded tweet instead?
https://platform.twitter.com/embed/Tweet.html?id=1674865731136020505
i don't particularly know how nitter scrapes, but i'm able to see tweets and profiles in my browser by setting the useragent to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Someone has reported that this trick does not work anymore. Fun fact: it was the same trick that was used to get the pre-Polymer UI back on YouTube, when it was still available.
I don’t know how long this “temporary” measure will last, nor if it will be temporary at all. In the worse scenario, it was a real pleasure to use Nitter, @zedeus ! Let’s stay tuned, but until facts get me wrong, I consider both Nitter and Twitter are dead from now on.
Thank you!
(Now, if you want to do a Nitter clone for Mastodon, to be able to browse it without JS needed, I won’t say no to this)
This could definitely work. Problem is that we still have to get the tweets from a specific user.
Could possibly scrape this for user timeline, although it only has the last couple of tweets.
https://syndication.twitter.com/srv/timeline-profile/screen-name/TwitterDev?showReplies=true
note the Android app still works without login, which means the "anonymous flow" is still possible:
https://github.com/4cq2/mech/tree/main/twitter
im having success scraping the embedded timeline as suggested by @fREAST
https://rss-bridge.org/bridge01/?action=display&bridge=TwitterBridge&context=By+username&u=DailyCaller&format=Html (1 hour cache)
[0] code for the temp fix:
github.com/dvikan/rss-bridge@c4a82ee3baIs this true? I can't check because I don't have a twitter account hehe
true https://platform.twitter.com/embed/Tweet.html?id=1674865731136020505
You can in fact live without Twitter, you should give it a try
Libreddit isn't using the Reddit API. it's scraping via public information. Unless Reddit decides to lock you out unless you're logged in, Libreddit will continue to work
Absolutely! in fact this is what i do daily.
However, there are still certain things that i need to use twitter for in my work, such as inspecting certain tweets from services that i use. Or just watching around friends that did not made the move just yet.
That's not to mention that, for nitter in particular, you can easily use a twitter account as an RSS feed for brands and services that don't have any other ways of communication.
Yes, a life without twitter is possible. Which is what Nitter mostly stands out for. Refraining from using twitter as a main tool and instead as an assistive thing.
This would be easier if people stopped using twitter to share current news which the MSM is entirely incapable of reporting on accurately.
You can live without all that stuff too. Stop reading news
@tuxpizza False. So much, that even the current maintainer is fearful of Libreddit's Future at the moment
"Ignorance is bliss" (If only bad things went away when you didn't know about them)
You know less when you read the news though
Can you stop with the lecturing? It's asinine.
Ahh I guess I was mistaken. I thought it was similar to Nitter but they are using the API but with an "Anonymous" endpoint. I'm betting on Reddit removing that aswell. Sucks.
How can I mark this as spam/off-topic? The report button is missing from the three dots.
That's because this isn't reddit
IT'S DA SCRAAAAPEEEERZ
using it for multi-account feeds also works* in a limited manner, when not receiving instance limited/bad gateway errors of course. :/
*edit: seems it's no longer the case.
thank you to everyone developing + working on nitter all this time.
accounts can be deleted and have limited access, which increases the complexity of the code
Requires a phone number with tor and having an account at all reduces anonymity.
Is it possible to have an anonymous phone number?
Obviously a borderline illegal phone account spamming implementation isn't likely, but thanks for nitter so much zedeus!
a simple bot detection script can check the xCsrfToken used in the search form of any nitter instance
after that the account is banned by twitter and the bot is taken offline
changing ip is easy and cheap, but creating verified accounts in an automated way is complex and costly
virtual sms number companies usually reuse the same number and each number can be used to create only 10 different twitter accounts
after any suspicious activity the account is blocked and the only way to unblock it is with a new confirmation of the phone number
the access volume of a bot also tends to be higher than a regular user, this draws attention in the same way that logged in users are often blocked after any suspicious activity
On the smspva.com website:
You would need the phone number longer than 24 hours. Basically the lifetime of an account. And thats IF your account isnt flagged for "suspicious activity", which is likely the case for new social media accounts. All new accounts usually have to pass some sort of cleared actions in a certain order over a significant amount of time before your account would be considered "real".
archive.is still works. perhaps they have a pool of logged-in accounts for scraping purposes
I get this when trying to sign up with tor.
Does this also require a payment method which reduce anonymity again....
Lol you must be new to scraping. Just because you verify your account with an sms once doesnt mean they wont ask again. :)
yes, and with the same number 🙃
In conclusion...
About 2(using accounts)
Since this change is intented for blocking access of scrappers, If some random account(verified or not) access lots of tweets/account in short time, they might decide that account is scrapping tweets and require re-verify of phone number, captcha, etc. No one knows since It's 100% up to twitter(or elon)
yes, the most logical way is for nitter to support login data or cookies like youtube-dl and yt-dlp do
in any case nitter will have to be rewritten to support these features
I'll note here that the Twitter rate-limit currently seems to be set incredibly low. I'm getting 429 errors browsing my timeline totally normally on my real account.
No, residential, accessing using browser without proxy/VPN. According to headers sent by Twitter the rate-limit is currently 50 requests to the TweetDetail endpoint per 15 minutes. Loading a tweet or loading more replies to a tweet both hit it.
https://platform.twitter.com/embed/Tweet.html?id=1674942336583757825 Elon says it will be unlocked soon, I really hope so.
thank you for the update. alternatives like bird.makeup, nitter, et al. are so important to have. thank you again to all developers.
temporary measure, just like the special 3 day military operation in ukraine.
Use this amazing nitter service since the beginning. It was so easy to read tweet, it was my daily reads. Without it, I wouldn't have read so much on twitter and I wouldn't have accumulated so much knowledge. Nitter will miss a lot for me.
its so over bros
Doing it hours before the end of the month after Musk has had a history of not paying contractors/suppliers screams that a major host for twitter is shutting them off at the start of July and "oh shit we have to turn stuff off or we'll not be able to keep the site up"
@Nitrousoxide
There might be something in there. Just as of now Twitter web is also down.
not just Twitter Web. iOS App also hit.
"rate limit exceeded" since +20 minutes, can't refresh timelines any longer
I think at this point it can be said that Twitter is too unstable to build any kind of software for it.
Even the people who are paying 10k's $ per month for api access get a completely unreliable and buggy service, and 0 support by Twitter.
for you enjoyment:
Everything is fine
Or it means that the android app by default tracks as if you were logged in.
What do you mean when you say "as long as you dont care about privacy" with regards to old.reddit?
The principle of Android app anonymous browsing is to register a hidden account with
_LO_prefixscreen_namewhen it is opened for the first time.And ONLY replacing the bearer token and endpoints will NOT solve the problem.
Is it any other way to browse accounts which I'm blocked without create another account? (which is IMO stupid) 🤔
I wrote myself this script: https://greasyfork.org/en/scripts/469910-twitter-com-to-embed
It rewrites pages to their embed widget URL, which doesn't need login. The relevant URLs to use yourself are
https://syndication.twitter.com/srv/timeline-profile/screen-name/USER(replace USER)https://platform.twitter.com/embed/Tweet.html?id=TWEET_ID(replace TWEET_ID with the number of a tweet URL)Those could potentially be used to fix Nitter as well?
Can confirm, the above is working well for both tweets and tweets > user feed
User feeds with the above method don't seem to be working for me, all just show "Nothing to see here yet".
For those who use rss feeds. When you go to https://rss.app/rss-feed/create-twitter-rss-feed and enter a twitter page URL you want to follow, you can get its content. You simply need to copy the URL and put it in your bookmarks. If you actually pay for the service, you can get the real rss feed though. I only follow 2 people, so its viable for me.
Maybe try to use ipfs to scrape in the user end and store some pieces of content on the nitter webpage, like
https://twitter.com/elonmusk/status/1675187969420828672
@yingziwu use this link instead https://platform.twitter.com/embed/Tweet.html?id=1675187969420828672
I don't have much more to contribute here, but people wonder how I've lived so long?
My desire (and track record) of outliving everyone I have a grudge with, serious or petty.
Cannot recommend it enough.
Really now... I have to imagine that 6000 tweets could be used up loading more/hitting the back button if you're in an area of particularly high engagement (politics, meteorology/local news, active hobby discussions).
And in addition, paying users barely getting anything more than the quote-un-quote freeloaders that were quote-un-quote 'data' 'pillaging...'
This man really is sui generis.
That script works great thanks.
With those restrictions from Musk, I can see a mass exodus from Twitter.
reading the comments/replies to musk's announcement was enough to hit 600 posts, my account is now frozen after about 5 minutes
this is a nightmare for any person with ADHD 😱
I guess advertisers pulling out from the platform was more catastrophic than what was implied in the couple interviews with him I've watched. "Running out of money" to pay necessary infrastructure looks like a believable scenario to me at this point since, well, 2023 mega-corps and ad agencies aren't really on-board with current twitter management and its "controversial" policies.
But on the flip side, instagram seems to be doing just fine sitting behind an authentication wall, so what do I know.
FYI TweetDeck works without any "significant" problems.
users on my fedi instance confirming that it does in fact have a rate limit. i wonder if it's 300/600/6000 per app (website, tweetdeck, phone app) or what
I tried to like a tweet but it didn't work, but they load
Musk has posted again...
https://platform.twitter.com/embed/Tweet.html?id=1675214274627530754
I do have 7 accounts on my TweetDeck maybe I have 600*7 tweet loading limit, can’t really confirm.
Yep, I can’t like either.
why does he even bother, that change doesn't do anything to address the problem
the site is completely unusable now for any kind of regular user.
Bots can still post freely.
I think, it is a campaign to promote twitter premium and to stop AI learning.
If this doesn't get reversed within the next couple of weeks I'll switch to using embeds as an emergency measure, but for now I'm just going to wait it out.
I appreciate the support everyone, it's amazing to see how many people use my little pet project. Nitter isn't going down without a fight.
Apparently older ios client is exempt from those limits? Maybe it's possible to figure out something from that?
https://www.reddit.com/r/jailbreak/comments/14o1v64/tutorial_no_new_twitter_updates_rate_limits_with/
This might actually really be temporary. According to this guy, the code that was pushed to block non-users actually caused Twitter to DDoS itself: https://sfba.social/@sysop408/110639435788921057
This guy doesn't know what he's talking about.
rip nitter
In fact, Twitter does not work at all:
Clicking on the links to Twitter opens the window "sign in to Twitter". Thus, this social network has turned into something like the Chinese social networks where they require wechat or something like that, only in the case of Twitter just nothing works.
This issue could also be called "R.I.P Twitter".
and Musk has posted again.
https://platform.twitter.com/embed/Tweet.html?id=1675260424109928449
I don't wanna post offtopic but i just loaded this tweet in normal twitter and loading the replies made me hit the new limit. This is comedy gold.
Maybe not of any use, but this extension is helping tweets load on chromium based browsers for logged in users. Perhaps this extension's source code could lead a clue? https://github.com/dimdenGD/OldTwitter
I don't see it mentioned but tweetdeck for some reason lets you to view posts and comment when above the limit, you cant see profiles and their contents tho, only your feed.
data from the last 48 hours from https://ciscorank.com/
@0xAB51NTH since github has no rate limit you can ctrl+f tweetdeck and see it was mentioned many times... only a few posts above yours
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
@mthmcalixto
Also using syndication addr is already discussed here several times...
Any fork that takes the data?
It seems like there's still a way to see tweets without an account.
When posting a Twitter link on discord, there used to be a preview but it stopped working as soon as they made the change, I guess discord crawler is considered as "not logged in". However, when posting a link preceded with "vx" (i.e. https://vxtwitter.com/status/xxxx), there's still a preview. I'm not sure how they achieved that though.
This... this is beautiful 🥹
Returns
Actual full result : https://bin.socialspill.com/ariruqiq.json
The Musk strikes again. I'm genuinely surprised that Tesla and SpaceX are still afloat with such a moron of a CEO.
At this point I start thinking that creating not so sophisticated Twitter alternative and deploying it to the cheapest DO droplet would be a viable business plan.
I dunno, it's just frustration at this point. A whatever-billion-dollar company can't pay for it's hosting, bruh.
Actually I tried that half an hour ago and it didn't work either. But somehow, Google is still able to get new tweets to the news feed. Sus.
That url is used in the embedded widgets, which continue to work (for now).
Tweet
Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/embedded-tweets/overview
Example: https://platform.twitter.com/embed/Tweet.html?id=463440424141459456
Profile Timeline
Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/profile-timeline
Example: https://syndication.twitter.com/srv/timeline-profile/screen-name/TwitterDev
List Timeline
Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/list-timeline
Example: https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/national-parks
Tesla is on its way
There are competent people (executives etc.)between him and day-to-day operation of those two companies. With Twitter it seem he is shouting into senior engineers's ears.
Twitter can also just allow the traffic only from the well known IPs of google bot. It's publicly listed
I mean... It's totally an option for them
Yep! got it working like this also.
Grab the content of the script tag, do some decoding and return it as JSON response to my request...
Dirty prototype in node-red you can import in to visually see whats going on:
Google needs to start applying their own rules to these big sites.
If the average site displays different content to Google and to users, it gets penalised/removed from search results. They must do the same thing to Twitter and Reddit. Hit them where it hurts.
They were crying because subreddit blackouts made search results worse, that's surely never gonna happen.
I think one of the staff or Elon is reading our comments lol
bird.makeup still working
It is trivial to completely defeat scrapers with no funding that power nitter. Nitter operates entirely at Twitter's pleasure.
Now appears to be gone as well. Not working anymore.
It started working again for me just now.
EDIT: It stopped again. It fluctuates :\
That particular endpoint seems to be appearing / disappearing in the last few hours. I'm guessing someone's actively working to remove it.
RIP syndication 🪽
guilty? 🤦♂️
@sunlightspectre
Totally agreed. Login-walling even to a public post (set public by the user) is the most stupidest move I ever seen. Because it is no longer public, this won't be visible to search engine indexing/crawling. This means any user searching for X won't be seeing any twitter posts on, say google search results. Tweeters will now have a hard time gaining an audience, especially artists.
googlebot have free pass to twitter
@devgaucho can that be used to bypass login walls as well as archiving?
Guys the syndication.twitter.com embed method works again
Thanks for the information.!
This is probably a dumb question, but what if a list has a space in the name? I can't get my list to show up this way. I tried a dash (-) an underscore and a regular space/ %20 and it all shows up as the list not existing. Not really sure about the rules about case sensitivity as well. Is there a maybe a way to extract the proper list name and user name from the lists' ID?
When you hover go into a list on the website, there's an ID showing, simply replace the id instead of trying to find a working list-name. You of course have to match the screen-name associated to that list.
Are you sure? I tried it with the example national parks links, where the list id shows up as 715919216927322112 (i assume this is the id, am i correct? The lists url is https://twitter.com/i/lists/715919216927322112 so i assume the number is the id) but
https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/715919216927322112
doesn't embed the list.
I'm pretty sure im being an idiot here, but i don't get it.
EDIT: Ok i just tried it on a random list instead of the national-parks one used in example and copying the ID does work, thank you so much <3
Yup it doesn't work for this one but I tried for one of my list and it weirdly it works
For example I clicked on this list on the website : and then created this link : https://syndication.twitter.com/srv/timeline-list/screen-name/JeandronMathieu/slug/1434116675968831497
And it works. But I agree, sometimes it doesn't. Not sure why.
Tried in on a different list and it worked just as you said, sorry for being dumb, thank you Write. Weird it doesn't work for the list used in the example, though
Edit: Aand it's broken again, it loads the list correctly (as in, it reads the list name and author) but doesn't load the tweets, as if the list was empty. Elon must be playin whack a mole with us.
Also it seems you can quite easily get your embeded link with the use of this page : https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fi%2Flists%2F1434116675968831497&widget=Timeline and just pasting your list url.
https://cdn.discordapp.com/attachments/874455648883052544/1125145584798548119/image.png
"Parse HTML with regex" I feel attacked
Has anyone built a tool to get an rss from a syndication link?
@TempUser13 dvikan has linked to rss-bridge in this issue 🙈
https://github.com/zedeus/nitter/issues/919#issuecomment-1615253896
Looks like this instance (rss-bridge.org) is modified to be able to parse the syndication (git.twitter-fix-2.be59a1e).
I am running my own instance of rss-bridge with latest docker image (git.master.d8bc015). This is not working for syndication (yet).
the twitter issues might resolve itself by its own so im waiting a bit before merging to master.
do you want it merged asap?
Thanks for offering, but I am not in a hurry. I switched a few of my RSS subscriptions now to rss-bridge.org instead of my own instance.
Having this fix merged would just be of interest for me, if also media content would be available via syndication, which seems not to be the case. Most of my subscriptions contain media, so this workaround will not work for me, regardless if it is on your or my instance.
Fingers crossed!
I wonder, will guys from https://github.com/DIYgod/RSSHub/issues/12732 switch to syndication too?
I used their instance for more than a year already. Currently some of feeds are working (probably cached), others are not.
OK so this is a source for the tweets. But you have to deal with that, know how to use it and how to integrate it to Nitter. And if the whole Nitter community use that source, not sure it will be working for a long time... due to a massive traffic increase.
you can add "showReplies" parameter to the syndication endpoint and will show the user tweet replies also
https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true
then to get more info about a tweet with just the tweet id you can use
https://syndication.twitter.com/tweet-result?id=1675214274627530754
Sometimes that's not even a bad option 😉
Syndication sometimes works, sometimes doesn't.
Right now, it is not working.
I'm getting this same issue. Thought my IP is blocked as I've been checking this link a lot, but VPN didn't help either. They seem to be on guard with this syndication thing.
Twitter employees are reading this thread for sure, this is when openness actually backfires
I guess at this point the only actual solution is to create *a lot of* accounts for readonly bots and deploy a load balancer so you guys wont rail the Twitter's potato servers
Do Nitter instances pay for captcha solvers?
I think the coward privated his account, too.
EDIT: Nevermind. The reason I don't see anything is because posts are hidden, but the syndication page itself is not.
I think Nitter can be saved...
Look at this browser extension that - while its advertised feature is to bring back the old Twitter UI - due to the "secret APIs" it uses, which seem to be something GraphQL-related similar to Nitter's internal API (I only skimmed trough the source code, haven't really read it), it can bypass rate limits: https://github.com/dimdenGD/OldTwitter.
As said by the developer (on Twitter), it needs an active login session. I tried logging into Twitter.com, installing the extension, and I think I've scrolled far more than the free 1000 posts I would be allowed to, so I'd say this works. If we can look at the extension source, we can maybe see what needs to be fixed in Nitter's source to make it work again (apart from requiring the instance admin to login with their own Twitter account on the Nitter server)
I'm sure they can see where the traffic goes and just block it, like they are doing with syndication now.
So any trick or loophole will be closed as soon as too many people start using it.
OldTwitter uses Twitter API 1.1 used by old TweetDeck. It's not secret at all and can be blocked like embed tweet
doesn't seem to work but thanks for the reply, ill try to selfhost with the specific commit, nvm syndication is broken i didn't see
Their crappy official API changes ruined my small project that checked 1 persons timeline every hour. Now I can't even check it at all.
The official API is completely useless at the free tier.
Wanted to use Nitter as a backup option, but looks like Twitter screwed that up too.
Hopefully they fallback on their changes.
good question, besides GoogleBot the twitter robots.txt still allows access from Slurp, Yandex, msnbot and bingbot
is it possible that it doesn't do ip verification on all these bots
but robots.txt seems to be outdated since it is still allowing bots access to /search?q=%23
Does anyone know how bird.makeup still manage to work correctly ? According to it's author it has 40% failure rate but that's still more than enough. https://sr.ht/~cloutier/bird.makeup/
His Twoot : https://social.librem.one/@vincent/110645408061048325
probably he uses twitter's api and makes a local cache to avoid repeating requests
even so he seems to be suffering with the platform's instability
confirmated.
https://git.sr.ht/~cloutier/bird.makeup/tree/master/item/src/BirdsiteLive.Twitter/TwitterUserService.cs
https://git.sr.ht/~cloutier/bird.makeup/tree/master/item/src/BirdsiteLive.Twitter/CachedTwitterService.cs
But that doesn't make any sense, you can selfhost it, there's no way he just share publicly his paid apikey .. ?
In his documentation you don't have to fill any apikey https://git.sr.ht/~cloutier/bird.makeup/tree/master/item/VARIABLES.md
He seems to use some sort of API 1.1 and / GraphQL combo :
git.sr.ht/~cloutier/bird.makeup@612637fdc7EDIT : Nevermind he seems to use some sort of API Key indeed, as you can see in the function
private async Task<string> GenerateBearerToken()insrc/BirdsiteLive.Twitter/Tools/TwitterAuthenticationInitializer.csOk this seems interesting, he seems he is able to get a Bearer tokens using built-in tokens from APPS as seens here
https://git.sr.ht/~cloutier/bird.makeup/tree/master/item/src/BirdsiteLive.Twitter/Tools/TwitterAuthenticationInitializer.cs#L36
He then picks one randomly and send a post request in GenerateBearerToken() which correctly returns a Bearer Tokens
You can try yourself with curl :
curl -X POST "https://api.twitter.com/oauth2/token?grant_type=client_credentials" -u "CjulERsDeqhhjSme66ECg:IQWdVyqFxghAtURHGeGiWAsmCAGmdW3WmbEx6Hck"You can then activate your fresh access_token :
curl -X POST -H "Authorization: Bearer your_access_token" "https://api.twitter.com/1.1/guest/activate.json"This will return you a guest_token.
You can then access graphql endpoints :
where x-guest-token is the token returned when you activated your Bearer.
You can get the user numeric ID from previous request, from twitter name : for elonmusk :
Then get all his timeline :
You have to replace the userID between the two "%22" tags. Here the user id (of elonmusk) is : 44196397
Which returns All his Tweets and Replies (too long):
https://bin.socialspill.com/ixomusor.json
To access detail of a specific tweet the url is :
Wow, this is a great find! So, can we just implement a Twitter API client and re-implement Nitter's fetching layer?
Updated my post, :) (btw, get rekt elon)
I guess the next step would be finding all the endpoints we can access to. I know the folks at Libreddit were able to do something similar by rummaging through a decompiled version of the Reddit Android client.
@Write https://github.com/zedeus/nitter/issues/919#issuecomment-1619067142 Is this not limited?
Well guessing that there is, hence why birds makeup has 40% failure rate and rotate between keys. Gonna let you try :)
It seems the Bearer Token is the same for all, so the first step can be skipped. But the
guest_tokenis different for all.So probably only this one can get limited. Maybe it is simple to just recreate a new one each few calls.
One limitation, according to tests with curl, is that each second or third curl answer is slow (~ 2 to 3 seconds).
You gonna have to make your own tests. I just showed an example as how bird.makeup works. I don't know anything more, sorry. For me the Bearer returned is definitely different based on which credential I use.
Yes, it is based on the credentials. But since they are constant, I got the same Bearer as you have shown above. All others probably, too.
There's a possibility they change about some times. I don't know.
Yeah, this app does not work. But you can try the curl codes above and it works. There is also not much room for twitter to prevent this without blocking their own official apps or force users to install new versions.
The problem with the code above is that it is less legal than the previous approach that was easier to get from the browser, I think. But not sure.
My goal to find all possible endpoints is starting to go well; just found my first API endpoint:
Here's all the operations I was able to extract from a decomp (using JADX) of the Twitter APK (version
9.95.0-release.0): operations.txtIt would seem that the Android and web clients use different queries, as I was unable to find any @Write mentioned.
It looks like Twitter's Android app makes use of Apollo, so there's a chance we may be able to fully extract all info about the schema from it; it will just require some reverse engineering.
this is a hard coded persistent graphql query:
Here's how you can get Elon Musk's profile info using the Android API:
It stops working fast.
That's expected and normal you just have to refresh your guest_token.
Poor man's nitter:
🤣
poor_man_nitter.sh
muskie shows up
Love your poor_man_nitter.sh
Damn . SO syndication isn't working- there's no working way for the plebs to read a TL is there (other than these shell scripts you wizards are toying with now) .. correct?
(no rss-bridge either)
hahaha
what if @ElonMusk monetizes scrapping for ChatGPT+ providing an improved API instead of chasing users?
UPDATE: apologizes for the off topic - won't bother you again.
Not that I found
@polkaulfield have you find a way to get only tweets without replies ? or at least that it returns more tweets, because for example, for elon we can't see any of his tweet because of all his replies
Idk, ill take a look tomorrow. I just automated your commands and parsed the output for the laughs. Amazing work finding this API :D
I tried the rIIwMe1ObkGh_ByBtTCtRQ/UserTweets to no avail.
It always returns The following features cannot be null: [then it list all features].
Not sure what / if I did something wrong.
Eliminating x-guest-token header has been working for me
Edit: Added authenticate(), TweetDetail()
Has anyone figured out any graphql endpoint for retrieving tweets by id? I have a list of like several thousand tweets I'd like to get asap with as little manual work as possible.
@devgaucho a la putcha, tchê.
@rupakhetibinit thoughts and prayers
@animegrafmays thoughts and prayers
Syndication API still works for that:
https://api.twitter.com/graphql/XjlydVWHFIDaAUny86oh2g/TweetDetail?variables=%7B%22focalTweetId%22%3A% + statusId + %22,%22with_rux_injections%22%3Atrue,%22includePromotedContent%22%3Afalse,%22withCommunity%22%3Afalse,%22withQuickPromoteEligibilityTweetFields%22%3Afalse,%22withBirdwatchNotes%22%3Afalse,%22withSuperFollowsUserFields%22%3Afalse,%22withDownvotePerspective%22%3Afalse,%22withReactionsMetadata%22%3Afalse,%22withReactionsPerspective%22%3Afalse,%22withSuperFollowsTweetFields%22%3Afalse,%22withVoice%22%3Atrue,%22withV2Timeline%22%3Atrue%7D&features=%7B%22responsive_web_twitter_blue_verified_badge_is_enabled%22%3Atrue,%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue,%22verified_phone_label_enabled%22%3Afalse,%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue,%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse,%22tweetypie_unmention_optimization_enabled%22%3Atrue,%22vibe_api_enabled%22%3Atrue,%22responsive_web_edit_tweet_api_enabled%22%3Atrue,%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Afalse,%22view_counts_everywhere_api_enabled%22%3Atrue,%22longform_notetweets_consumption_enabled%22%3Atrue,%22tweet_awards_web_tipping_enabled%22%3Afalse,%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse,%22standardized_nudges_misinfo%22%3Atrue,%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse,%22interactive_text_enabled%22%3Atrue,%22responsive_web_text_conversations_enabled%22%3Afalse,%22longform_notetweets_richtext_consumption_enabled%22%3Afalse,%22responsive_web_enhance_cards_enabled%22%3Atrue%7DThis works for me for UserTweets
I dumped json to file tweets.txt rather than parse. Got it from here before they switched to using user's auth tokens:
https://github.com/Spicadox/auto-twitter-space/blob/master/index.py
Updated: https://github.com/zedeus/nitter/issues/919#issuecomment-1619298877
Loving your poor_man_nitter.py
Working for me thanks a lot.
Working thanks a lot !
@ScamCast you should add this one :D
btw guys, just changing the bearer const. is enough to make nitter work 100%
Here :
github.com/zedeus/nitter@dcf73354ff/src/consts.nim (L5C21-L5C21)(Just figured there's all grapheQL endpoints here too)
Proof : https://nitter.socialspill.com/I really don't think this method (which comes from bird.makeup) will hold long time.
[guys, don't even try to scrape this instance, please, I will shutdown it soon anyway]
EDIT : Well it already stopped working, 🤡 not sure if it's nitter not refreshing guest_tokens or something else
I dont' know enough about nitter guys, it stopped working even though it's not my ip that is limited since poor_man_nitter with the same Bearer tokens still work. So... No idea.
All graphql returns 404 not found on nitter
It just stopped, it doesn't work.
Yeah, I know, I edited my message. But it worked for a while. And I really have no idea why it doesn't work anymore since with the same ip / bearer it still work just fine with the poor_man_nitter script
the strange thing is (well, strange to me, a simple end user; i'm sure the rest of you lot will understand) it didn't just stop... it went from displaying all tweets, to just pinned ones, to 502 errors for any additional accounts looked up (the previously looked up still displayed pinned tweets only, even after reloading in a private browser window), then 503 errors for all.
thank you kindly @Write for this latest effort.
Pretty sure we need all endpoints compatible with this Bearer / APps for it to work properly. For now
For now we have some working, such as pNl8WjKAvaegIoVH--FuoQ/UserTweetsAndReplies.
We would need all of them that are compatible with this Bearer / Apps to make it work and replace all the one ntiter currently use :
github.com/zedeus/nitter@dcf73354ff/src/consts.nim (L5C21-L5C21)For example one of the most important is SearchTimeline, which we haven't found one yet compatible with this specific Bearer.
Inspecting all the requests that the Twitter Android app is making inside an emulator. Getting lots of useful stuff.
Twitter uses some funky domain for their app API.
https://na.albtls.t.co/graphql/3JNH4e9dq1BifLxAa3UMWg/UserWithProfileTweetsQueryV2Greetings,
I am no way a coder/developer but have been using Nitter for a while and sad to see the recent developments. While looking for solutions, I came across this (I remember using it some years ago). Wonder how it works and if it may be of any help at all:
https://tweettunnel.com/maxabrahms
We can see the tweets, albeit in limited forms and clicking on links does take us to Twitter (and no, I didn't sign in).
My 2 pennies.
Best
Just in case someone don't know about this: RSS-Bridge (official instance and other public instances) also supports generating rss feeds from other sources, e.g. public telegram channels. It's not the 1st time twitter breaks 3rd party program, so I'd assume it's not the last. Some organisations have telegram channels in addition to twitter accounts, e.g. some news sites.
It is sad that things turn out this way. I avoid tiktok, facebook and related services due to their gross privacy violations and not wanting to support their unethical behaviour. Now others are having accessibility issues.
Been using my own local Nitter instance to generate a combined RSS feed of users I want to follow, because fuck using the cluster fuck that's Twitter and their algos.
Anyway, ever since Musk-chan put up the login wall and limits, it stopped working. But very rarely, it will still work and let through a couple of tweets. 4 hours ago, one of my rss feeds suddenly contained 14 new tweets that were made today.
It's interesting behavior but briefly looking over the ~500MB logs didn't make me any wiser. Other than seeing a whole lot of 404s for "UserByScreenName" and ratelimit errors for "https://api.twitter.com/2/timeline/profile/"
Try this one:
G8jKRx5LiyrRDs5FcsUjsw/SearchTimeline
@Write @ScamCast I found a ton of GraphQL endpoints in the android app. Worth looking:
https://gist.github.com/polkaulfield/8c84087fafa476e86f768ad25ec44f7b
https://gist.github.com/polkaulfield/adfd371a5157c2fb4b334244b29e094a
Have you tested them? The the Android Bearer does not work and the endpoints are device-dependent. The Mac Bearer works.
Inserting the above constants in the nitter source code makes only the user search work, for me.
Was able to find all these from the Android App. Have all their parameters too.
Taking a long time to put them in a readable format. 😪
Can you connect using the guest token? I found a ton of hardcoded endpoints too inside the app.
Didn't test, but that's what i found looking for
com.twitter.graphql.GraphQlOperationRegistryNice! I just went through and parsed them all out
https://gist.github.com/ScamCast/2e40befbd1b61c4a80cda2745d4df1f4
Could OldTwitter be used as reference?
I've hacked together an Express server that fetches recent tweets using this approach and produces a JSON or RSS feed. PRs very much welcome 😄
https://github.com/12joan/twitter-client
please take a look at #925. it rotates bearer tokens through a list of hardcoded ones. it would be good to add the macos one if anybody has it.
i found that with the session cookies of a random user the instance mostly works (#830 is needed for that). if there is a rate limit refreshing a couple of times usually gets you a different bearer token and resets them
Hello everyone, nice works of you guys while I was asleep, so cool to see such a community.
Well, the creator of bird.makeup seems to have one available :
https://git.sr.ht/~cloutier/bird.makeup/tree/master/item/src/BirdsiteLive.Twitter/Tools/TwitterAuthenticationInitializer.cs#L36
Just do
curl -X POST "https://api.twitter.com/oauth2/token?grant_type=client_credentials" -u "3rJOl1ODzm9yZy63FACdg:5jPoQ5kQvMJFDYRNE8bQ4rHuds4xJqhvgNJM4awaE8"to get the macOS bearerAAAAAAAAAAAAAAAAAAAAAIWCCAAAAAAA2C25AxqI%2BYCS7pdfJKRH8Xh19zA%3D8vpDZzPHaEJhd20MKVWp3UR38YoPpuTX7UD2cVYo3YNikubuxd
You could also add the iPad and iPhone one as I see thoses are missing
I personally tried multiples SearchTimeline endpoint you guys posted but it always returns
{"errors":[{"code":126,"message":"Logged out."}]}for meI'm giving up here guys, I don't have the faith to remove SSL Pinning of iOS app and finding all the endpoints for all platforms.
Good luck everyone and thank you ❤️
Curious as to what @zedeus thinks of all this
Not sure what this is about, do be cautious
@coda101
Seems like
tweettunnel.comis using ads whichbcloudhost.comis part of.Ghostery will block these two attempts when visiting
tweettunnel.com:httpx://www.bcloudhost.com/9c88e091a401e3c00259b11fe2e91264/invoke.jshttpx://likenesscollecting.com/be501a9bff1a8da595943ddced3825ed/invoke.jsChecking both
tweettunnel.comandbcloudhost.comthrough PaloAlto urlfiltering (https://urlfiltering.paloaltonetworks.com/query/) brings following result:URL:
tweettunnel.comCategory: Social Networking
Description: User communities and sites where users interact with each other, post messages, pictures, or otherwise communicate with groups of people. Does not include blogs or personal sites.
Example Sites:
www.facebook.com,www.twitter.com,www.linkedin.comCategory: Low Risk
Description: Sites that are not medium or high risk are considered low risk. These sites have displayed benign activity for a minimum of 90 days. The low risk category includes both sites that have a history of only benign activity, and sites found to be malicious in the past, but that have displayed benign activity for at least 90 days.
Example Sites:
www.google.com,www.schwab.com,www.amazon.comURL:
www.bcloudhost.comCategory: Grayware
Description: Web content that does not pose a direct security threat but that display other intrusive behavior and tempt the end user to grant remote access or perform other unauthorized actions. Grayware includes illegal activities, criminal activities, rogueware, adware, and other unwanted or unsolicited applications, such as embedded crypto miners, clickjacking or hijackers that change the elements of the browser. Typosquatting domains that do not exhibit maliciousness and are not owned by the targeted domain will be categorized as grayware.
Example Sites:
`
Easiest way to block access to
www.bcloudhost.comandbcloudhost.comwithout involving a browser extension is to modify your /etc/hosts file (similar exists in windows too) and point both domains to either 0.0.0.0 or 127.0.0.1 or 192.0.2.1 or some other non existent IP-address (as in your browser will not get anything bad from it).Edit: Made all domains non-clickable.
When/If nitter resolves the current issue, is it
nitter.netwho will get the first commits or do there exist some devsite who is public which runs the latest code?@12joan Any chance you're publicly hosting this? Sounds like we need to test for rate limits 😉
Special thanks to @Write's information. I've made the Twitter route of RSSHub works again. It always re-authenticates to get a new session after being rate-limited. It should work well unless this approach or the server IP is banned - the latter may occur on the public demo, deploy a personal instance to get rid of it.
https://rsshub.app/twitter/user/elonmusk
https://rsshub.app/twitter/user/elonmusk/excludeReplies=1
https://rsshub.app/twitter/media/bugcat_capoo_tw
https://rsshub.app/twitter/keyword/RSSHub
@Write What surprises me is that the
keywordroute implementing search failed to work in my local tests (403 Forbidden), but it turned out to work well on RSSHub's public demo - I didn't change anything of the route except credentials. It seems thathttps://twitter.com/i/api/2/search/adaptive.jsonworks in specific conditions which are currently unclear, after being authenticated with your approach.github.com/DIYgod/RSSHub@7e45fe12e9/lib/v2/twitter/web-api/twitter-api.js (L106-L114)Anyone making a tool I can save to the wayback machine (WBM)? Complete with the proxy's html featuring links to the original resolution of images and gifs (because I saved only those so far), as well as the tweets? I'm still waiting of when the WBM is able to bypass this.
Saving twitter pages directly now results in "Unverified web crawlers are not allowed" rather than a 404, and according to this: https://twitter.com/kevinmarks/status/1676140926077468672 also blocks googlebot.
nice work @Rongronggg9 👏, can read tweets by user id?
@devgaucho how do I access older tweets so that I can save them to the WBM? It looks like it only listed the most recent X tweets rather than extending to the first existing tweet. I have a bunch of tweet URLs and how do I make rsshub view a specific tweet?
I don't think it works on the demo server right now.
anything other thanI found some keywords like "nitter" that work, but I think SearchTimeline with session cookie works more reliablyRSSHubkeyword does not work for me.I found that oauth credentials sent to /oauth/token always return the same bearer token, so i opted to hardcode them in #925
in that PR, everything but search works unauthenticated (if you refresh often enough and hit the right bearer token) -- the rest can be made to work by adding a cookie header in config. it seems that per-user rate limits get reset if one resets guest-token, even if the cookie is the same
embed url https://web.archive.org/web/20230704212606/https://platform.twitter.com/embed/Tweet.html?id=1675260424109928449
The demo is behind a load balancer, so each request may be processed by a different server. Still, I am quite unclear about the conditions to make it work.
@devgaucho
Thanks
Awesome work !
Can you enlighten me on the differences between the /i/api/2/search/adaptive.json endpoint and the (for example) G8jKRx5LiyrRDs5FcsUjsw/SearchTimeline endpoint ? Can you provide a CURL example of usage with /i/api/2/search/adaptive.json ?
Also your issue seems kinda related to the one I had when I replaced the Bearer in a Nitter instance, worked perfectly for some times and then only 404 without me being able to understand what was wrong since I used the same Bearer / IP. Anyway.
Also it seems that your /excludeReplies=1 endpoint only shows one tweet for elonmusk, certainly because you're filtering entries from /UserTweetsAndReplies or something like that, but @worstperson (https://github.com/zedeus/nitter/issues/919#issuecomment-1619332131) actually made /UserTweets endpoint works and you can get last 20 tweets without any replies :
I've updated @ScamCast's awesome poor_man_nitter.py to add UserTweets endpoint.
what's funny is, if "withV2Timeline": False, then the API does respect the "count": 40, variable !
I set it to false only for the endpoint I added (/UserTweets). I don't really see any downside to it for now.
Executing above code like so :
./poor_man_nitter.py | jq '..|.full_text?|select(length>0)'Returns magnificiently 40 latest tweets just fine (as it's the default set in the python file), NOT cropped or anything.
I tried to change "count:" variable to an arbitrary high number in UserTweets and can get UP to 155 latest tweets.
I was working on a full python module. Should be able to release it tomorrow.
Never too late :)
Has someone made any endpoint work with an other "auth" than the iPad one ?
during the rate limit time, I never saw the rate limit error on my iPad and Android phone. Tweetdeck and the web client immediately died. Also leads me to believe that the rate limit is not based on username otherwise my tweetdeck being open would've affected my iPad's Twitter which it didn't.
In my latest commit, I used the
consumer_tokenandconsumer_token_secretfrom the Android client, but after I got the bearer token, I further got the open account. it is not clear what is the difference between the open account and the bearer token.Except for the
SearchTimelineendpoint that needs to be re-adapted, other endpoints can be used directly, endpoints used for the community has became login required.Sorry, I made a mistake. In fact, the public demo is behind a load balancer and some nodes are configured with developer API keys. So in some cases, the search result is returned by dev API. See also: https://github.com/DIYgod/RSSHub/pull/12754#issuecomment-1621087622
I am not sure if Nitter will get a new session automatically after receiving an empty 404, that may help. Your IP may still be banned after requesting a lot. The public demo of RSSHub and my instance still work pretty well though.
Nope. It uses the
/UserTweetsendpoint and does not apply a filter. It is probably because the load balancer leads you to a node configured with dev API as I've said below. Try/excludeReplies=1&forceWebApi=1. There may be some bug while fetching elonmusk's tweet, try other accounts.But for me, it is interesting to know that the v1 API is still functioning. AFAIK new API keys have no v1 API access and are applied unusable rate limits. But it seems that existing API keys with v1 API access have not been revoked?
Thanks to every contributors for the work made !
Can someone say how far we are to make nitter work again ?
I didn't understand much but thank you for the help and warning about fishy stuff.
I found that the rate limit of using the existing
UserTweetsAndReplies/UserTweets/TweetDetailendpoint directly will become very low (50 times per 15 minutes), updating to the v2 endpoint used by the application will increase the rate limitUserWithProfileTweetsQueryV2/UserWithProfileTweetsAndRepliesQueryV2/ConversationTimelineV2(500 times per 15 minutes), but the search endpoint did not change it's rate limit.Tweets can be viewed when logged out again. Website uses the
/2ICDjqPd81tulZcYrtpTuQ/TweetResultByRestIdendpoint instead of/NmCeCgkVlsRGS1cAwqtgmw/TweetDetail. Replies can't be viewed, only the requested tweet. Rate-limit is 50 tweets per 15 minutes, but it's using guest tokens so token refreshing should work.Dropped by nearly 50 places by today. Good job Elon-guy, keep playing stupid games
things are changing
you can now view individual tweets without an account: https://twitter.com/elonmusk/status/1675390796718014464
the individual tweets are without replies
you can even view the users account if you click their username within an already opened tweet
but they will not have a feed
if you try to go to a users account directly it will hit you with the login page
because of the above it appears nitter is slightly less broken
instead of an error you can view accounts but they are feedless: https://nitter.net/elonmusk
adding to this:
you can now view twitter feeds if you are using the googlebot useragent
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Just to note, you can see the last few recent posts on the feed, but you can't scroll and can't see comments. Still better than nothing.
they fully undo'd the login-only change by now.
But Nitter still doesn't work apparently.
Not fully undone. Watch a tweet on https://twitter.com/"user"/status/"tweet_id" while not logged in works fine. Anything else (user timelines, search) still is redirecting to login dialog.
syndication is still down https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true
this is a good signal 🙏
I'm rather more pessimistic, I think they are gauging how much they can squeeze without getting hurt. They may be satisfied with this or little more, which would be pretty terrible for instances that don't use api (all except privacydev?).
except for profiles
I'm still unable to read profiles or fetch RSS feeds on nitter.
Removing the woke disease from social media cupolas has its consecuences. Musk is now paying for it.
I'm seeing TLs again on instances..
edit
Meh. Maybe it was just timing. Not working.. Rate limits..
zuck show your magick stick to musk, now he has a new, even bigger problem to face
Sorry, I join now the discussion.
What modifications are needed to make Nitter work?
bypass or integrate the authentication to the crawler
Tks. What kind of interventions are on the code?
the code needs to be rewritten to incorporate authentication cookies generated somehow
some hackers are already using mobile app visitor authentication code to do this
I think simulating the native twitter web authentication is the safest way to make a crawler with long-term support
the simple cookie incorporation works at youtube-dl and yt-dlp, but for public nitter instances a cookies rotation with many different accounts credentiais is need @nicfab
Is there any intervention on the code you can show me?
not, @zedeus is waiting before act in nitter, but tweet tunnel and rss hub are already working
Tweettunnel is, rsshub seems to only show paying users
rsshub was actually only showing the profiles anyway
but they are getting tweets from non-paying users too
FYI twscraper seems to also be a tool that is able to rotate between accounts.
that's the right way 👍
lol, didn't know
It seems like we won't need that after all, they removed it. Strangely Nitter still isn't working though.
Twitter gave back the ability to see individual tweets without logging in. However you still can't see whole threads, or all tweets for one account.
Any idea how to make the
countparameter stick onUserWithProfileTweetsQueryV2? OnUserTweetsyou can do"withV2Timeline": falsebut it doesnt work on this endpointIt's probably still worth exploring/implementing as many contingencies as possible. Just like some alternatives frontends for reddit using their private api, it makes it so twitter can't break nitter without breaking their own functionality. Mutually assured destruction
Maybe so, but I would still keep those efforts in a separate branch, just in case they should ever be needed.
Or they could eventually be patched-in, as an "emergency" sort of back-end, should this happen again (Assuming the Twitter API isn't going to be completely blocked-off in the future).
Si vis pacem, para bellum
This intance seems to work also for hashtags (maybe only for this moment?)
https://nitter.x86-64-unknown-linux-gnu.zip/search?f=tweets&q=%23riptwitter&since=&until=&near=
I've been thinking more about how to prevent companies from doing this sort of thing: making content publically available, then putting everything behind an account wall after gaining enough users. LinkedIn did it, Instagram did it, Facebook did it, etc.
Even having decentralized frontends isn't really promising, if these companies will just keep changing their backends. When we update the code, they can keep changing it. It's how bibliogram ended up dying. That was also decentralized like nitter but just for instagram but the original author closed it down because it became too much to keep up with.
The only real way to do this is to create accounts with different IPs and devices and make the companies think they're real accounts (e.g. let them age, add a phone number, verify via email, hell--even set up 2FA) and start sharing these accounts in a distributed database or in some centralized repository. the data for accounts can be encrypted with keys that only we have so companies cant see what accounts they are. That way, it would be virtually impossible for companies to stop. If people are interested in this idea, I don't mind starting another thread to discuss more.
Just lets hope this isnt the next level of escalation:
https://commsrisk.com/ukraine-police-raid-simbox-farms-spreading-russian-propaganda-via-1-5-million-fake-internet-accounts/But instead of a propaganda farm its just a nitter cluster to bypass the moody behaviour of the Twitter CEO.
it's my instance. it uses #925 and #830 on a burner twitter account. i've used it extensively over the past week and it has worked fine so far. some spurious rate limits, but they disappear on refresh (since that triggers token rotation)
So this means Nitter can work again with current situation. Let's hope main code will soon integrate these fixs.
those patches require you to register a twitter account to set up the instance. i think it's the best solution so far, but it's really not great.
i think a true zero-config solution would use Android's reverse engineered APIs, aka "guest accounts". this would require a working oauth2 implementation for nimlang, or at least the request signing part. and it's a lot of work to implement.
it's smarter to sit this out as zedeus does
https://alternativeto.net/news/2023/7/twitter-quietly-removes-log-in-requirement-for-viewing-tweets/
hmmm
thanks a lot for your work!
I just spinned up an instance with #925 and #830 to see how well it holds : https://nitter.socialspill.com/
EDIT : Well I seriously down't know. It stopped working after a few minutes only. 🤷 Only 404 errors
refresh a couple of times? it should fix itself after that. make sure you got the cookies and csrf config right (only ct0 and auth_token)
Well yes, I just checked again the website and nitter.conf does contains the right cookieHeader with correctly filled ct0 and auth_token. I also filled xCsrfToken but the value seems to be the same as ct0 value. I just built the docker container after pulling branch, I kinda don't get it. I can refresh / reboot / delete the image & container and rebuild the image, and it stays at 404 answer from twitter.
which url/path/view are we talking about? i cannot access the instance at all at the moment.
Sorry, I was rebuilding the image yet again just to "see" if I missed something. It works for now. Will see how much time.
It already struggles again, for example on your instance you can load
https://nitter.x86-64-unknown-linux-gnu.zip/search?f=users&q=%23riptwitter
and then click 'Users' tab multiple time, or just CTRL+R / F5, really (which makes the URL load each time), and it works everytime. whereas on mine :
https://nitter.socialspill.com/search?f=users&q=%23riptwitter
If you do the same thing, 50% of the time you get blank results.
EDIT : Well 75% failure rate now 💀
nah, I also have this on my instance (there's just a caching proxy in front of it). it is probably because there are some bad tokens added in #925 that don't actually work. haven't spent the time figuring out which one it is, would need to A/B test it. my guess is that the ipad token is the most stable one
ok thanks.
All I can say is that when I tried all the ApiKey / endpoints, only the iPad one worked for me with this script :
(haven't checked nitter's endpoints with it)
Well it's down : You can see it only shows empty pages. Twitters returns only 404
twitter implemented authentication in their apps.
they're definitely watching this issue.
It was mentioned previously, and the endpoints gathered so far is from the mobile side (which aren't working anymore).
there was an update to android yesterday, but at least the old android apk still works (
Twitter_9.95.0-release.0_Apkpure.apk)yep it's busted. was a fun 48 hours or so
yes to be clear i am saying the old android apk i still have still allows me to view profiles and search tweets without an account
Yes hahaha
I baffled with space karen logic here ,if there is no way for soon-to-be user to view what are on twitter how it suppose to survive, this is not a porn site?
Probably a way to manipulate the stock market with a hit and run (even if TWTR is delisted since 27 oct last year such numbers can have a value for a potential new owner or against competitors)?
"Look we have 250 billion users!" even if there are only ~8 billion people on this world and give or take about 60% of them use internet on a daily basis.
The 250 billion would come from burner accounts now needed but the investors wouldnt care about facts when they see a fast way to earn money.
Question is what really happend at Twitter when their CEO had a bad weekend and went mayhem (but for this perhaps a separate thread should be created)?
If there were some massive AI scraping going on then why not just temporarily block or throttle those IP-addresses and call it a day?
Thats what I do when I see Google, Bing, Yandex and the others from time to time misbehave during their indexes of sites I host.
If next level of throttling is needed then give official Nitter instances their own API access (token key + IP-addresses in combo) where the official Nitter instances have a throttle on their own towards the enduser (as in whats normal for a Nitter user (based on IP-address/cookies) to consume in terms of requests/day and then add a margin to that to not get throttled unnecessary).
Using caches at Nitter would also lower the volume of requests towards Twitter if thats the issue now when they canceled (and signed a new contract a few months later?) their hostingcontract with Google Cloud according to:
https://www.techradar.com/pro/twitter-has-apparently-started-paying-for-google-cloud-againSad 😢 but maybe it's time to spend more time to build on an open decentralized protocol like nostr
There is already ActivityPub. And the problem is, that 90-95% of people on Twitter will still use it. Why ? Because of the large amount of users on that network. On any other network, they will never have half the audience they have on Twitter.
So Nitter or other ways to scrap Twitter content will remain usefull for a while.
I feel like I might be shooting myself in the foot with this, but https://tweettunnel.com/ is still working (more or less).
Anyway, I just wanted to thank Zedeus for his work on this project. It has been ridiculously useful to me over the last year. Thanks to Nitter, I've been able to maintain my main source of information on certain very niche topics without supporting Twitter's bad practices. Shame on me for waiting for the platform to go down before thanking you for your efforts. Better late than never. Cheers mate.
That's pretty cool and will keep this in my back pocket, but not being able to see comments, or add people to a following list without logging in still stinks. For me, I've been following the fritter evolution from fritter, to quacker to squacker and gives you the ability to browse twitter like that old app, and hopefully again one day in the future, actually have a feed for users you follow.
As far as I know, you're the second person to ever make an instance using PrivacyDevel's NSFW workaround. I've been waiting for someone else to use it. Thanks so much for making that happen!
this is not working.
you probably have a cache on your device that still allows it.
it definitely still works on a hard-reset android device. 9.96 does not work, 9.95 does.
Sportsbots for Mastodon is still working.
Well you're welcome but as you can see it's not working 🤷
i wonder, have you made any modifications such as adding/removing bearer tokens? I tried out removing the ipad token and it broke a lot of stuff. on the other hand, using only the ipad token breaks tweet search, as far as i can tell. made some updates to the #925 recently
Nop didn't made any change to the Bearer.
Maybe that's because I tried to pull and manually merge both #925 and #830 ?
I deleted everything and directly cloned https://github.com/drpepper66666/nitter/tree/twitter-token-rotation now, building the docker image right now, will see how it goes.
the login cookies are unfortunately still a requirement, but starting a new branch sounds good
maybe @PrivacyDevel has the time to maintain a proper fork of this with everything merged together
If I clone your https://github.com/drpepper66666/nitter/tree/twitter-token-rotation it's not enough right ? I have to add #830 changes then
I've seen this from all ends and I seem to be the guy who comes up with solutions but after seeing fixtwitter and TweetTunnel gave me an idea. Knowing that tweet-tunnel is going to start experiencing heavy ai traffic and to help lighten their burden and recover all the tweets made from before 6/30/2023. What if we asked TweetTunnel to host a part of their database through a torrent and then that way some of their traffic will be lightened because that has seemed to be the original problem in the first place from the main site. This way we will be able to preserve and recover what may have been lost or is currently lost due to these new changes by musk. this way all we will have to now request is the media from fixtwitter and then problem solved for all past tweets. I don't have a solution for future tweets but that seems to be the best idea I have for old ones...
I have been using https://nitter.net for months since twitter got annoying about asking me to login before being able to see more than a handful of tweets. I refuse to create a login as I don't want them tracking me and I only use twitter to keep an eye on a handful of people.
Needless to say I have been unable to do so since the recent change. I have been keeping an eye on this and have seen some progress from errors, to users not found to most recently at https://nitter.net/MarkDavis showing everything except the tweets.
Will it soon be possible to see the tweets as well? I hope so, as my alternative is to convince the people I follow there to post elsewhere as well or in place of twitter.
Nitter was very important place to me.
I was given a lot of detail information by it.
I'm very sad...
I've been thinking the same thing, but instead of torrenting I would use ipfs which allows you to share the tweets individually via http
twitter's data is gigantic, in the order of petabytes, someone could create a kind of decentralized scihub to share the scraped tweets
the individual tweets shared throught a ipfs collaborative network can be indexed using a central website with a free api open source
but this is just a dream 🥺
Then exclude videos and compress all other things
is 1.5 zib only for the status messages
That many Tweets would mean an average of
...Tweets per human currently alive on Earth, which seems excessive to me. Is it possible that the majority of Tweet IDs are unused?
Not trying to weigh in on whether this is a feasible idea or not, just curious about the numbers. 🙂
Edit: Tweet IDs have been timestamp-based since 2010.
sad to say @thechesu
atm I start to think that any app that rely on a third party service is a no go most of the time !
@devgaucho Your calculations are extremely flawed.
Tweet IDs are not (directly) sequential, and I can assure you that there is no way that a quintillion tweets has ever been made. These are Snowflake IDs, with a timestamp + machine/worker ID component.
You also have to consider that most tweets are nowhere near the (relatively new) 280 character limit; the old limit was 140 characters.
A quick Google search will tell us that:
Some quick and very inaccurate calculation with our new figures, without accounting for UTF-8/emojis, t.co links, images/videos, time of day, growth rate, spikes during disasters/sports/viral events, spam, database/indexing overhead, other data fields, replication, and everything in between:
I think that my calculations are way too low (for uncompressed, unindexed text)... if anyone wants to waste some time calculate a more accurate number, feel free!
yes, great new: not all tweets were created by humans obviously 🤖👈
the number is correct, the tweet numerber is a interger with auto-increment
this is the oldest teet available (number 20)
can you assure me how, voices in your head? wishfull thinking?
it's not timestamp, they are sequential numbers
yes, the oldest tweet is the id 20, if you don't believe me check for yourself here 👉 https://twitter.com/jack/status/20
for timestamps you would have to use microtime as the volume of new tweets exceeds the barrier of tweets per second easily
i put 280 to normalize as an intermediate number, i know well that there are millions of tweets with less than 140 characters and others with more than 280
as nobody knows how much exactly musk intends to expand the maximum tweet size and they are only accessible for verified counting so 🤷♂️
280 characters is currently the minimum for storing all tweets posted by unverified accounts until musk decided to increase the limit to who knows how many characters, so yes, I defend this value as the basis regardless of the actual value
in case you missed it I took into account the multibyte characters, so I multiplied 280*4 🙃 easy like cheese
your calculation is much more rough and inaccurate than mine because it takes into account numbers that you don't have at your disposal, obviously it is much better to estimate upwards than downwards
this is classic occam's razor case:
if you want to go ahead looking for horns on horse's heads or microtime on integer numbers with auto-increment go ahead 🦄
@devgaucho They are not "auto-incremented" IDs, they are autogenerated Snowflake IDs just like @signalhunter explained. He linked to the wiki page, and here's Twitter's own page explaining it: https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake
half-truth, I was right about the auto-incremented ids from 2006 to 2010 🤡👍
looking at the snowflake format description with the same recent tweet:
this make sense, sorry @signalhunter and @zedeus 🙏
If we just let big tech do whatever they want, eventually we will lose access.
The long-term solution is to establish media campaigns against big tech's trespass into our personal lives by requiring login. Society-wide surveillance is trespass into our personal lives.
The reason that we care about privacy at all is that trespass is wrong.
It's part of a wider strategy popular among corporations, where they act as middle men between buyers and sellers and squeeze both.
Monopsony from the persepective of suppliers, oligarchy from the perspective of consumers.
It does not violate anti-trust laws because the chicago-school interpretation of anti-trust is now dominant, which requires clear consumer harm, rather than uncompetitive behavior.
The best solution is to stop using proprietary social media platforms and invest in a free decentralized platform that can't be easily censored or monopolized. IRC is still an example of this...
Unless users want to take their activity elsewhere the escalation ladder of censorship and social manipulation will only continue.
Not to play devil advocate here, but it's probably time to let twitter go.
The move to require an account makes good sense from a business stand point. It is mainly to twart off AI scrappers and letting them have access to all Twitter's data for free while on top of that adding server costs.
Imagine a future where you don't even need to go on any website anymore because whichever AI you're using has access to everything?
Example: Hey ChatGPT/Bard, "Show me all today's tweets from Elon Musk".
With that, you don't need to go on Twitter anymore and that's bad as it kills their revenue. And this is just Twitter as an example, but it can be applied to any websites, especially news outlets.
The fact that once again, most of the internet can be centralized to whatever AI you're using is alarming. Google is probably the most at risk here if they don't quickly polish their own AI. Any AI will basically replace your search engine.
As time goes on, we can probably expect more and more public information become private (walled off behind accounts) to fend off the AI leeches.
Account required is just a logical business move. It's not some vendetta to force people into making an account. If some people don't have a Twitter account by now (I'm one of them), then it's a firm decision and they probably won't make one now either. And even if some do, it's a drop in the bucket.
Maybe if AI wasn't around, Elon would have still made the move, who knows?
What we can learn from this is to not use any free or paid centralized service anymore where the user is the product.
We can give Twitter a little credit as in, they are so big today because they're the ones who came up with the whole Tweet concept/idea, they were the 1st and it got mass adoption.
If Mastodon or any decentralized equivalent were there before Twitter, maybe Mastodon would have been the one to be mass adopted.
Remember that Twitter is a convenience, not a necessity. If Elon decides to make Twitter $5/month for everyone tomorrow, would you continue to use Twitter? Probably not. Then it's not a necessity.
I'll be moving to ActivityPub/Nostr, whichever ends up harnessing most users, instead of playing cat and mouse with Elon.
This won't take care of the AI problem, but at least, it puts the users back in control of their own data.
I wish Nitter and Twitter the best. Big thanks to Zeus and all the other Nitter contributors.
I hope you can still find a way to access Twitter publicly.
This isn't even close to how faddish generative ML tools like GPT process information.
We can stop using them but politicians, Government agencies, news outlets and businesses have replaced their public information front-end communications increasingly with Twitter exclusively. For example, politicians communicating directly with their constituents, weather alerts, local news and information, public transport information and service updates plus increasingly customer service representatives and service outages.
These are not trivial or non-essential uses for social media, they actively prevent people being disenfranchised from their democratically elected representatives and society. And these groups are not simply going to use decentralised social networks or give up the cost-effective solution of passing the costs of hosting and maintenance to Twitter. In the same way YouTube content creators are not going to give up the free hosting and bandwidth of their content.
I totally agree with you. I'm not expecting the whole Twitter user base to migrate overnight, or even migrate at all.
But I think it's not too far-fetched to speculate Twitter might go under at some point. And if it did, even though the government seems to heavily rely on it as you pointed out, they will probably not bail Twitter out.
They could get bought up by a competitor, but not sure how it would fare with the FTC.
If Twitter truly goes under, the user base will have to find another place. Hopefully not Thread 👀
But the decentralized alternatives are currently free. Server instances costs are on the shoulders of their administrator.
If they become too much, the admin of an instance could just close registration and users could then register on another instance.
I can't really predict how well ActivityPub/Nostr instances would hold up with a sudden influx of a massive user base such as Twitter's, but to re-itterate: users do not bare the hosting/bandwidth costs on decentralized services either.
According to some political theorists, platforms such as Twitter should be held in collective ownership, either by employees, users or by governments.
Never too late to start caring about this stuff in the "real world".
I have a set of fixes that restores all functionality, including search. It should be ready later today, unless it's too unstable.
Never trust proprietary softwore or services, they will soon, or later, always become like this. We, as the user, are useless if we have no interest for them, that's naturally as a company. Don't cry, boys. It's always been like this.
Funny enough for someone to be a masochist who wants to receive whatever the shxt is given away from those whoever control the services. Then, shut up and directly use twitter, why'd you go here? I know this is not the place talking about this. But I know, there must be lots of people who want to say shxt about this change of twitter.
@stopmotio You're right. I should talk this on other forums instead
@stopmotio Different bearer token, new endpoints, and tweaked parser code to accommodate for the minor changes in JSON structure
Regarding the GoogleBot method: Twitter is actively changing the HTML output. I implemented a full parser that worked perfect and today it is partly broken because Twitter changed the
metatag from start tag without end tag to self-closing tag.Also, it seems that they limited the number of Tweets from 12 to 8 and changed some urls from absolute url
https://.../sufixto relative url/suffix.It is government regulations that create and enable big tech in the first place. Without government regulations, youtube would not have access to cheaper servers and cheaper internet lines. Without government regulations, amazon.com cannot stay huge. The only reason that amazon.com has huge competitive edges over small retail stores is government regulations that squeeze small retail stores and help amazon.com grow.
(Big) government creates and maintains big media, big tech, big police/military state, big education, and everything bloated.
Big tech and big media, in turn, help the government by promoting government regulations that squeeze independent businesses.
It is a giant web that depends on each other and reinforces each other.
Nitter is a band-aid that might stop working eventually unless people address the worldwide web of power.
We're back for now. https://nitter.net is up and running with an experimental patch that provides all previous functionality, except search (I have a potential fix, needs more research), without suffering from the awful rate limits imposed on the previous endpoints.
The code is on a separate branch: https://github.com/zedeus/nitter/pull/927
Wow! Excellent! Thankyou! However, the RSS url's don't seem to be working.
For example:
https://nitter.net/AlessioUrban/with_replies --> WORKS
but
https://nitter.net/AlessioUrban/with_replies/rss --> "NOT FOUND"
Is this expected behaviour at this stage?
Confirmation that Nitter’s working again from someone who’s been lurking here since day 1 and doesn’t know how to code.
@level323 Yes, I disabled RSS for the time being to reduce load.
Fair enough. Thankyou again for your efforts.
AUD 30.00 donation just made via librepay as a gesture of thanks for your work and in the hope that RSS capability is back up soon.
Thanks for the (relatively) quick solve!
Will the age restricted tweets also come back?
Thanks! https://nitter.moomoo.me has also updated.
It works! Many thanks!
I've checked all the instances and updated the wiki to show which instances have updated. I also removed instances that don't have a dns record anymore. Please update the wiki once you've updated your instance!
I guess you can use the PrivacyDevel-fork for that.
Yesterday I created an instance with that (on Docker) and it worked well. But then my instance became rate limited now.
I guess the recent changes from here have to be included in the PrivacyDevel-fork.
A big "thank you" to all the people helping to solve this issue!
@zedeus and all the other contribs, thanks a lot for your work :)
Any idea on when the RSS wiil be back ? Is it rather a matter of hours, days ou weeks ?
Is the RSS disabling a matter of nitter.net or it's hardcoded?
Is there a sort of nightly docker image (latest-devel or somesuch) that has the fix?
I'm glad to hear Nitter's functionality is restored! 😄
For those asking about RSS, we're seeing some good results over at 12joan/twitter-client. PRs welcome if you find any edge cases that don't work correctly.
@woj-tek you should be able to tell Docker to build
https://github.com/zedeus/nitter.git#elon-fixinstead of using the image to get the PR.@level323 thank you for the support!
@Snyret "sensitive tweets" access will not be restored beyond @PrivacyDevel's experimental account PR for the time being. I haven't made any decisions yet, so far it seems a bit risky and unstable.
@moom0o thank you! With over a hundred public instances it's quite tedious to update it, I really should start working on an instance site à la Searx/Peertube/Invidious.
@edel79 RSS works perfectly fine, I even made it so the new replies page with reply threads only includes tweets from the account rather than all the ones being replied to. I chose to disable it using the configuration file for nitter.net while everything is being tested in production. I'd like to see it run smoothly for another day or two before opening the floodgates again.
@woj-tek it's just nitter.net, see here: https://github.com/zedeus/nitter/blob/master/nitter.example.conf#L25
@zedeus thanks for the great work, it works again !!
One little thing, it seems that you added a more-replies-thread class which didn't exist before.
Any way to remove those and see replies not elipsed ?
thanks !
@zoubidabida with the new endpoints used Twitter includes these "reply threads" or "conversations", I'm simply parsing and rendering them. When a reply thread, or "conversation" has more than 3 tweets in it Twitter doesn't include them, so I need to display the "more replies" link similar to how Twitter does it.
Had a random thought and I figured I'd leave it here.
If scraping Twitter isn't easy anymore, what if someone made a chrome extension that people could install, and then the chrome extension passively scrapes tweets and forwards them on to some sort of Nitter API. Then Nitter could serve tweets from it's own database, and rate limiting wouldn't be an issue because you'd be just getting passed data for free from people who installed this extension.
You could even keep track of who reported the most new content and give them points for bragging rights, or do some sort of points/crypto incentive. Like the points/coins can be used to buy ads on Nitter, and the people who scrape new content are rewarded in these points (so they have an incentive to install the extension and not just rely on volenteers)
Anyway, just some thoughts.
That is a good idea. We are already using this method for one of our applications. We need a lot of tweets for different purposes. We have a data entry team that uses this extension and sends the tweets to the server. This same idea can be extrapolated to be used with Nitter (it needs to be thought through if the extension is going to be public without any auth). I'll be more than happy to open-source my extension (although it's very basic at the moment; just reads tweets, but can be easily updated to incorporate all the other data).
@lukeprofits
There are quite a few problems with this approach.
is there any way to view a locked account on nitter, when I have access to it with my twitter account?
Point 2 is a major setback.
Set up your own instance with the cookie data of your Twitter account?
@saqib-ahmed @gfpj the problem with point 2 is indeed authenticity. How do we know a tweet is authentic and not manipulated, or entirely fake? How do we prevent garbage data? I'm not sure, maybe a verification system that requests clients to independently request and verify data, but that opens another can of worms (e.g. the same problem as Bitcoin's 51% attack where enough bad actors verify incorrect data maliciously). This just turns into distributed systems security 101, which is not an easy problem to solve. I would've loved to set up a distributed cache for all instances to use and contribute to, but it's infinitely easier said than done.
Here's the attack I was thinking about that makes this a very difficult problem to solve: https://en.wikipedia.org/wiki/Sybil_attack
If we assume that viewing of any individual tweet will NOT be blocked for guests (by using any available API such as preview embedding), then the only valuable information – is the list of tweets ID, with respect to author account.
So, nodes should transfer only the list of pairs (tweet_id,user_id), but not tweet contents itself. Then, any caching server could download those tweets on its own from Twitter.
The reason to send user_id is for simplicity of handling the protocol and storage (like
{user1:[id1,id2],user2:[id3]}) so we can get both the author and the date of each tweet (since the date is a part of id).UPD: probably, quote, retweet and reply fields might be useful also. Or not, depending on available info in single-tweet guest downloads.
I don't think any are unsolvable.
If the user is logged in and using Twitter, they will be served tweets. These are not hard to scrape. (the idea is incentivise Twitter users to "share" the tweet info with us without us having to sign in, so that people can use Nitter without Nitter needing to scrape.)
They would have to send the tweet ID, and what it said. Since people would be reporting everything they view, it would be easy to confirm that what one person reported is accurate. If the ID and text match, and you've had x number of reports, it'd be very likely to be correct. Also, if you catch a user reporting BS, then they can just not be considered a trustworthy source. This would work well (at least for decently popular accounts.)
It would miss small obscure content, but if you 100k+ real followers, pretty likely that this would get most of everything.
All six of the "Unofficial Bird" servers have been updated and are now running the #927 branch too:
https://unofficialbird.com
https://canada.unofficialbird.com
https://india.unofficialbird.com
https://nederland.unofficialbird.com
https://singapore.unofficialbird.com
https://uk.unofficialbird.com
tweet_id and user_id can be changed by twitter, better generate a cid of each post
[nitter.lacontrevoie.fr] is also up to date now.
I knew about changing of display name of a user, and inability to use ID of the user in search queries: https://github.com/mikf/gallery-dl/issues/2624#issuecomment-1140021564 – which creates mess, but partially solvable by tracking renamings of each user (by their id which is permanent. Right?)
I never heard of tweets changing their ID. Can you elaborate?
@moom0o
Don't update the wiki by saying that all the instances are outdated if the fix has not even landed into the main branch...
You can do it once the fix in the main branch and that most maintainers can easily update to the latest fix.
How to build this branch:
elon-fix?It seems stable enough, I'll just merge it now and push search if I get it working. Anything is better than the current state where nothing works.
@zedeus + rest of the community: keep up the great work! :-)
@aleksusklim If @zedeus decide to create a giant base of tweets between the instances, then twitter can make a change and destroy all base to prevent nitter be strong.
Thanks zedeus and the community for the great work,
I've updated too : https://nitter.socialspill.com
What change? Destroying tweet_id and invalidating all of existing /status/ID links that were posted anywhere ever?
Thanks so much for these changes - my local nitter instance is back to normal and getting RSS feeds :)
In a perfect scenario supplied content could be verified using TLS.
https://tlsnotary.org/
This wouldn't solve the problem. Nothing short of an external Nitter-controlled server spying on and validating Twitter's SSL handshake from the client's perspective would prove that the data is genuine—way too complicated to be worth considering, if it's even possible. It certainly can't be done from JS.Correction: I took a look at that link, and it looks like they actually managed to make this work. https://github.com/tlsnotary/PageSignerThe normal web sandbox (without extension privileges) can't interfere with or spy on the TLS process. I'm still not entirely sure how they managed it even with an extension; it looks like a custom TLS implementation running on an extension-controlled TCP socket.
If we're going to continue discussing distributed Tweet exfiltration, I think we should do so in a separate thread so that we're not spamming people who are here to hear news about Nitter. 🙂
Isn't that extension just parsing HTML document? Why bother with TLS?
@MrBoombastic It's using a secret sharing technique with a trusted third-party server to cryptographically prove that the data originated from the HTTPS server the user claims it originated from. Way beyond the level of cryptography most developers (including myself) are familiar with.
New thread to avoid spamming this one
Are we talking about @saqib-ahmed's extension? He didn't say anything about cryptography. Assuming that "data entry team" means people, who can be really trusted - the extension can be sending Tweets as is, without verifying anything.
do you know a graphql endpoint to retrieve a user's "following" data?
First order solution is a reputation system: tag each database submission with a "submitter ID" of the Nitter instance submitting the Tweet. Allow other instances, or users, to flag garbage data. Then instances can decide which "submitter IDs" to trust.
(not discussed here -- how to ensure submitters don't cheat and spoof IDs)
Then one can do a 2nd order reputation system, where people assess others' moderation decisions.
Finally we provide overall security by allowing users to "vote with their feet" and choose the Nitter instances which provide the highest quality data. The system is designed so that malicious instances and database flooding don't compromise the performance of legitimate instances.
Sure we could also do some kind of crazy proof of work blockchain solution, but simply letting people choose which Nitter instances to trust is probably enough.
Without direct access to Twitter, the server instance cannot proof anything it its own. At least, Twitter can hire malicious nodes that will DDoS such caching server by flooding it with anonymous fake contributions. The scale might be so large that nobody could check or flag all of them manually.
Otherwise, WITH a direct access to Twitter (at least for individual tweets by id/link) – the server instance could trivially check each contribution (without re-checking already confirmed tweets; and banning IPs of clients who are trying to give many fake tweet_id in attempt to make the server hit other Twitter limits)
I don't believe Twitter will ever make its links (and thus, tweet_id) non-functional.
I literally figured this out independently and was coming here to edit my post to reflect this!
If it's not possible to check every Tweet in the database (it probably isn't) then it is probably enough to:
It breaks down, will you allow anonymous contributions or not. Currently it feels like they should be anonymous, just posted to some public endpoint. If you'll make this available only to "logged into Nitter" users – then nobody would do this, because "hell, I'll rather login into Twitter and just stay there! Why do I need to authenticate on some questionable scrapper, spying me on what tweets I look?"
Also, I don't see how an ordinary user surfing Nitter can check, "is the tweet valid or not?", because if I have an access to Twitter – then I probably don't need Nitter altogether.
Nope, I don't care much, and won't actually do anything for Nitter project.
@icy-ux @aleksusklim please move it to #931
Thank you for the continued work on this great project!
Glad it's working again, but when will searching for tweets be available again?
@sawtoothpuma Within a few days if it works properly.
@zedeus Good to hear. Thank you.
I've just pushed search to master, meaning all previous functionality is now working again. Let's hope things remain this way.
But no RSS on the nitter.net instance yet :-(
Any estimates on when RSS is coming back to nitter.net?
There's a strange memory leak that causes fairly rapid and steady memory usage growth, and enabling RSS would significantly exacerbate that. Once I figure it out I'll enable it again.
Thanks a lot.
https://nitter.socialspill.com/ up and running with search.
Search is back on *.unofficialbird.com too.
Many thanks!
Thank you for fixing search, but it seems that we can’t (on my favorite instance, at least) search for tweets posted before 1st July. Well, it’s still better than nothing, I guess.
Images are working again. (Had some issue with an AdGuard list wrongly blocking stuffs : https://github.com/blocklistproject/Lists/issues/1028)
Thank you for your work zedeus! https://nitter.x86-64-unknown-linux-gnu.zip/ now runs off of stock master branch, with RSS feeds enabled. Let's see when it goes out of memory.
Thanks Zed! With I tend to use nitter for a feed for many different sources, and i noticed I cannot get all my feeds to show at once it generates a "No items found" but if i remove some accounts it shows them all. last night I thought i had 12 working, today 19, but the moment i add 1 more to it then it fails.
I just tried using the top 20 twitter handles it worked. Top 25 it did not.
Is there a limitation in place for this, or is tied to the memory leak?
Is there any instance w/ RSS support back yet?
There we go again! 🚀🚀
Will this support NSFW or is support for that not in the cards anymore?
https://ntr.nah.re/ always had RSS enabled.
Only if you log in your T account into the instance config, as PrivacyDev did on his.
Any chance for search to give results older that the beginning of July, or must we forget about that because of the API changes?
Use the bearer token for TweetDeck Preview to replace the token for legacy TweetDeck (now).
It seems, the old tokens work again, at least with Fritter app it works again since today, without updating the app.
So it was exactly blocked for 14 days.
Same for me too. I have an old instance of Nitter running that I forgot to update ages ago (Used it for RSS), and it just a little bit ago it started spitting out a whole ton of new stuff. I guess it's all back to normal, now?
The web-based stuff works again, but is still significantly limited compared to the way Nitter does things now. The work was thankfully not in vain.
Uhh, https://nitter.privacydev.net/ is no longer getting through the age restricted wall. Did twitter upped their system?
@stopmotio
Did twitter voluntarily did that, or COPPA or any law actually prohibit bypassing age restrict wall?
I really hate that they rather age-restrict all the user's post when not all of them are NSFW.
@stopmotio is there another instance that still gets around it though? This list of nitter instances no longer works: https://github.com/xnaas/nitter-instances (404)
As a side note, yes, I completely despise any form of mandatory age verification. Login walls to just view content on a website or to download a file is already annoying enough. Having the internet needing your personal identifying information like an airport is excessive and potentially dangerous.
how do I do this?
down again @zedeus 😒
the problem is in reading the tweets of the profiles, the search continues to work normally in the version 2023.07.14-f881226 @zedeus
As a workaround, you can switch to the "Tweets & Replies" tab https://nitter.1d4.us/Twitter/with_replies
https://nitter.1d4.us/Twitter/with_replies/rss also works
------- Original Message -------
On Thursday, July 20th, 2023 at 11:15 PM, stopmotio - notifications at github.com @.***> wrote:
No “No items found” on ntr.nah.re, but RSS feeds are down for the “Tweets” tab (showing only a few years old tweets). Reported in #953, #961 and #962.
Deprecation of the /1.1/guest/activate/ endpoint in the Twitter API
This is bad news.
Hi @Write, do you know if there is a function similar to these that you have created to search for hashtags?
Thanks!!
Interresting to know why the "tweets and responses" still work. If someone knows...
The googlebot method was updated today (?). It now returns ~100 tweets, but the order seems to need sorting now.
Yes, but now its not chronologically. It looks like it only returns the 100 most popular tweets, e.g. the Fall Guys account almost only displays tweets from 2020 (when it released) and nothing from this year even though they tweet every day.
Yes, that was also my first guess. So, maybe new tweets are not shown anymore, assuming the user has already made at least 100 tweets :(
Update: it's even worse. Many Twitter accounts look like they have 0 tweets with the new googlebot method update.
So, it's not really usable anymore.
Since requests for tweets including replies still works, would it be possible to use it and filter (exclude) replies when rendering the html content of the "tweets" page ?
@edel79 kinda, but it could return 0 non-reply tweets for accounts like Elon that reply a lot. Not really a feasible option. I fixed it by using an older endpoint.
Someone to update once again this list with today's fix ? https://github.com/zedeus/nitter/wiki/Instances
For now all the 6 public instances I use are not upadated with today's patch.
Hello everyone,
Just updated my instance to latest commit : https://nitter.socialspill.com/
I've just enabled RSS, but I may disable it anytime since it seems there's a memory leak issue.
I have updated my instance (which I don't disclose much because I use it internally) to the latest version (2023.07.22-72d8f35), and it seems that tweets that are too old are not showing up in the "Tweets" tab.
For example, for this account (which appears to have ceased operations at the end of January 2022), pinned tweets are shown, but tweets from the end of January 2022 are not. However, they do appear in the "Tweets & Replies" tab.
A search does not seem to find this tweet.
I am using the 2023.07.22 version and the RSS does not work. Like test here https://bird.trom.tf/TROMSITE/rss I get User Not Found... Do I have to activate it manually from config somewhere?
EDIT: Eh maybe it works now...weird
The syndication api works again for timelines:
https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true
It shows the latest ~20 tweets and/or replies
Note:
showReplies=falsewill now switch to "most popular tweets" like the googlebot method and will also ship out ~100 tweets.Parsing the json can be done by searching for the first
{from start and the first}from end.Update (15 Aug 2023):
showReplies=truedoes not work anymore.RSS feeds stopped working for me for a few days, also with latest images.
https://<NITTERURL>/<USERNAME>/rssBut this works:
https://<NITTERURL>/<USERNAME>/with_replies/rssJust sharing for those who read by RSS as a temp 'fix'.
Oh, I wondered if the RSS problem was just me. I get 403'd.
Was about to set up a couple of RSS-to-Nostr bridges with rsslay before moving away from Twit- ... i mean X.
Any idea why it is broken or if it will be fixed?
That instance is unstable from time to time, I'd recommend finding a different one or setting one up for yourself.
It's not broken nor unstable, I disabled it to curb scrapers. It'll be enabled again at some point in the future, for now please use another instance.
I'll once again recommend 12joan/twitter-client if you need a quick Twitter-to-RSS solution until things settle down. It's a little easier to self-host than Nitter if you're using Docker, no configuration required. A few of us have been using it for a couple of weeks now without incident.
I noticed that sensitive content flagged by the user (not age restricted) results in a "this page doesn't exist", and is flagged as age restricted on https://nitter.privacydev.net/
Example:
https://twitter.com/canaaaaamin1003/status/1671702937272467459
https://nitter.privacydev.net/canaaaaamin1003/status/1671702937272467459
Logged out:
Logged in:
Nitter:
Today, on the instance I use (ntr.nah.re), “Tweets” tab and RSS feeds work fine again. Until the next break…
This instance works but sensitive content is still restricted. Has there been a solution found for that yet?
Well, it broke again…
https://github.com/zedeus/nitter/issues/983
the search for this fork still worked intermittently at commit 813a71e until 10:00pm on August 16, 2023
sometimes it returns nothing and the rss feed returns an error 502 (internal error), but eventually it works
from what I saw they use a proxy system with cache because old searches are updated, but new searches return as 404
game over? https://twitter.com/nima_owji/status/1691850092385370483
I hope https://twitter.com/xDaily/status/1691859235951583630
via dailydot
They use a login based alternative?
See commit
d5689f2musk magick formula = less captcha and more subscription
https://twitter.com/elonmusk/status/1691969296543711471
probably. from what I noticed they use a cache in the search system, the current lag is more than 10 hours
the cache prioritizes some profiles, the most accessed I assume
in July 21 they exchanged the endpoint of the profiles for the endpoint of the search system, I believe that this is the main source of the tweets that feed the profiles, so they simplify by focusing efforts on only one endpoint
as some delay but stil works
Which accounts are working for you besides "jack" and "elonmusk"?
I tried a few from https://en.wikipedia.org/wiki/List_of_most-followed_Twitter_accounts (like BarackObama, justinbieber, Cristiano, rihanna, YouTube) and all of them showed the same "An error has occured in one of your routes." page. All of them are visible without an account on twitter.com.
Or is this a result of the 10-hour cache delay you mentioned?
my personal account is accessible thourght the search box with a 2 hours delay
the cache is local, should prioritize the most accessed sites in the instance in case
The ones I checked all had a mix of random new and old tweets almost like "highlights".
apparently, tweets with more likes are being shown first (when viewing a profile when logged out).
https://9to5google.com/2023/08/16/twitter-profile-sort-like-counts-logged-out/
Superseded by #983
Hi,
For now the only instance that seems to be working is nitter.net. I usually use unofficialbird but that one--and many others--report the jester route exception error.
@rohitrsm83 the conversation continues here: #983
Thx