looks like X/twitter(?) broke something again #983
Labels
No labels
accessibility
bug
discussion
documentation
enhancement
feature parity
good first issue
help wanted
question
rss
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
PAS/nitter#983
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Also, the syndication api for
showReplies=truedoes not work anymore:https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true
but
showReplies=falsestill works, showing the tweets ordered by like count...https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=false
Yes, it is not working now. I hope the Nitter people fix this soon.
Is there a online/CLI tool converting
https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk
to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
Not really. showReplies=false shows years-old content when not logged in.
Down again...
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything?
We can just search for the first
{from begin and first}from end and then parse as json.If a user has not so many tweets (500-1000) then the chance is quite good that also newer tweets are within the most popular 100.
But of course it's not a very good solution. At least, nitter should have it as a backup when nothing else works, this method can be used.
Indeed
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job?
Not sure I understand ? It expose far more informations than needed and it does expose the date and all
Here's an example for one tweet only :
EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev.
Ok, thank you, I'll try this.
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august...
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets.
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing.
im using this for my bot and it working fine with cookies and headers.
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K).
is there any forecast for solving this problem?
Looks like https://nitter.privacydev.net/ is working
That one is a fork which uses account credentials. See #830
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user?
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them"
User feeds not working on this
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with
Strange. privacydev (without credentials) works more or less for @elonmusk, but not for other users like for instance @barackobama.
It could be cached tbf
Not logged in, pulling a TL in the normal fashion (eg. twitter.com/elonmusk) supplies results like this as well.
Has this always (recently, since all the major upheavals) been the case?
No. With a browser specific user agent up to a few days twitter/username would just show the profile and one tweet. Using "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" and coming from a tweet of the user the correct and fresh timeline was shown.
They patched it 😢
Yes. They switched the same time on the main page and the syndication api with
showReplies=falseto show the top 100 tweets. First, they changed the googlebot method from show the latest 20 tweets to the top 100. That was in the time, where twitter was completely locked without account and only googlebot method worked. When they found it a good solution to break scraping, they switched the other two to do the same. But since today, alsoshowReplies=truedoes not work anymore.Additionally, they removed timeline support completely on the "embed page". Thus, we can expect that timelines on the syndication api will go away completely within the next days or weeks (also top 100).
I assume, within the next 3 months, scraping without an account will not be possible anymore, besides top tweets. The top tweet method is a quite good solution for twitter: "normal users" without an account think Twitter is working perfectly fine, scrapers can't scrape anymore. Account-havers are not realizing any differences.
Same thing for I. On every device. I don't trust this Epstein and Nicole Junkermann affiliated Musk one bit.
First of all thank you for everything that's been done so far sedeus and every one who's been helping and testing, I wish I could. So sorry to post this if it's irrelevant, but just found it and figured maybe it's related? TweetDeck is officially becoming a paid service
While nitter is getting fixed I tried to visit the regular twitter url but noticed that the list of tweets isnt sorted by date anylonger but seems to be random and plenty of very old posts shows up?
It seems to be somewhat sorted based on number of likes (but scrolling down I see that the sorting is somewhat sloppy): https://twitter.com/elonmusk
Is this just me or did the sorting order break the other day aswell?
I found this alternative https://twstalker.com which seems to be working , can it be used to scrape data as well like nitter was used before?
Working Nitter Instances (As of posting):
https://tweet.whateveritworks.org/ (Rate Limited as of 16/8 14:54)
https://nitter.privacydev.net/
These instances currently only support tweet searching
@Bismanpal-Singh No idea how that one works, it's a strange shady closed-source site I thought at first it was all cached, but it has new tweets too. Might just be a huge account pool, or maybe they reverse engineered the guest account thing, dunno.
Fine , gonna be cautious of it for now , any idea till when we'll get nitter back up and running?
Nope, I don't have much hope and the only solution left is complex.
Shouldn't they be open-source as of GPL licensing?
All the best, hoping everything works out soon!
Don't know why, but https://tweet.whateveritworks.org/ works much better. https://nitter.privacydev.net returns many empty results, "Load more" returns nothing and many tweets found cannot be accessed.
@vulet Why do you think that https://twstalker.com uses the GPL?
Sorry that one doesn't look like Nitter, I didn't mean that one.
Nitter is working but in limited capacity, tweet searching and user searching works but profile viewing doesn't
Nitter works fully now, tweets are loading but requires a couple refreshes of the page. All nitter needs is a auto refresh system to keep fetching tweets when an error occurs.
still not working for profile viewing here
I can load profiles and tweets with multiple refreshes as well.
Can anybody indicate some instances which are working? The only one I found is https://tweet.whateveritworks.org and that's rate limited a the moment.
Some shorter search queries work randomly each ~8th time. Longer queries don't seem to work...
I just need to twitter user's latest tweet without delay.
I used
https://syndication.twitter.com/srv/timeline-profile/screen-name/XXX?showReplies=truebut doesnt work anymore. https://twstalker.com/ has delay. Is there any advice?Sadly not, unless you use the official twitter app without an account
Still not working here:
`https://nitter.net/MarkDavis/with_replies
An error has occured in one of your routes.
Detail: fatal.nim(53) sysFatal
asyncfutures.nim(385) read
asyncfutures.nim(385) read
asyncfutures.nim(385) read
asyncfutures.nim(385) read
asyncfutures.nim(385) read
asyncfutures.nim(385) read
packedjson.nim(294, 19)
falseunexpected end of object`Dave
yeah looks like its completely dead again
Sadly it is
Regarding the idea of users submitting their own account credentials that Nitter will use for scraping, how likely is it that the user could get banned? Is there any way for Twitter to distinguish between somebody actually using the official Twitter web app and Nitter making raw queries to a reverse-engineered API?
As previously mentioned, Twitter can check for the number of accounts associated with a given IP, but that shouldn't be a problem with self-hosting. Rate limiting shouldn't be a large problem for RSS, since you can just have your RSS feeds update less frequently.
This is risky, twitter can and will ban regular people for using their account as a scraper
Can somebody explain: Using https://nitter.privacydev.net I mostly end up on an instance which calls itself (upper left corner) "nitter (SFW-BACKUP)". Then nothing works. (The kind of "error" changes: Mostly I get "502 Bad Gateway". Sometimes "Tweet not found" or "No items found".)
A few times the instance calls itself "nitter (O2)". Then there is some chance that I can do a search and visit the tweets found. (Only some chance, because often it moves back to "nitter (SFW-BACKUP".)
It suddenly worked for a bit then broke again soon after, as my feed reader was briefly able to update my feeds.
Well, the good news is that even if your Twitter account is permanently suspended at least you can still plug your token into the PrivacyDevel fork. That worked just fine for me, because nitter only needs to view tweets. The rate limiting shenanigans made me go back to upstream without a token though anyway.
(My account was flagged for looking like a bot because I hardly used it in favor of nitter, I wasn't causing trouble grr)
I think that Twstalker only loads the first image or video in a post, so you'd still have to click through to the tweet to view the full content. So it's not optimal
Still better than:
Probably reverse engineering the OAuth signature which is used by the app to authenticate API calls when used in guest mode. Upon examining the requests sent by the app in this mode, I noticed that the OAuth signature appears to be a straightforward OAuth 1.0a HMAC-SHA1 signature, as the one shown here .
However they are probably using a guest consumer secret / OAuth token secret pair to sign the request which I was not able to find as, unfortunately, my expertise in app reverse engineering is limited. Is anyone aware of ongoing work in this direction or has insights to share on this subject?
Perhaps this scraper that I randomly found on reddit could be of use?
https://old.reddit.com/r/Twitter/comments/15sohmk/found_a_way_to_scrape_any_twitter_profile_without/
That tool uses https://nitter.poast.org
The twstalker page has 2021 as date on the page, maybe it's an old twitter app api that still works.
I think it is a MAC/Iphone/Ipad version or maybe some old hbbtv app or xbox/ps3 or so.
* I am not the owner of TwStalker
I wrote a post in my blog but not in English... so
TL;DR
Known issues
x-rate-limit-limitfrom response. If you request the same endpoint six or seven times (not more than ten) within a short period of time, you will lose access to all timeline endpoints, which will be restored after a day.Thanks a lot @BANKA2017, that's extremely valuable! I'm scared this method will break as well after being put into production, but it's totally possible large data mining companies are already using it, and Nitter wouldn't even make a dent. I'll look into implementing this in Nim soon and see where it takes us.
This was exactly what I was missing! Thank you!
However I just noticed that on version 10.2 (iOS) they disabled the option to browse the app without an account, hope this doesn't mean bad news incoming
HMAC-SHA1 is never really "straightforward", as you have to hash:
oauth_consumer_keyoauth_nonceoauth_signature_methodoauth_timestampoauth_tokenI would more describe it as "nightmare fuel". at any rate, here is a Go implementation if it helps:
https://github.com/1268/media/tree/main/twitter
Just tried it and It is giving Socket TimeOut Error I don't know why is there anything i should keep in check I am just copying this code into node and executing
I don't have an iPhone, but on the iPad version I always get redirected to the login page, so this might be a feature specific to the Android version.
It still works on Android so this workaround should still work
It was possible until version 10.1 on iPhone too, but not anymore since 10.2, but maybe it's just me
Good news everyone! I conclude that it is possible to easily acquire thousands of guest accounts within just a few minutes by using proxies, and they are all usable from a single IP address without getting rate limited. The rate limits per account work the same way as guest tokens, with a 15 minute window of x requests being allowed. It is therefore 100% feasible to get Nitter back up and running, it just requires a bunch of proxies. I will also develop a service that fetches these continuously, and lets operators request guest accounts for their own instances without having to pay for proxies.
Just to be clear on what a "guest account" is, it's this strange anonymous account old versions of the Android and iOS apps used to make when you opened them for the first time. It doesn't use an email address, a password, or even a customizable username, and they cannot be viewed anywhere or logged into. For all intents and purposes it's essentially a set of temporary credentials API that work for about a month. Is is therefore not equivalent to setting up a bot farm to create fake X/Twitter accounts, and it's not quite the same route as Aurora (the Google Play store that used a very large pool of Google accounts).
@zedeus I would advise not automating the process of giving instances access to your proxied guest accounts; verifying that someone actually has an instance up and running seems like a good idea to prevent abuse and DoS attacks on the proxy service
that would prevent private instances, that are not reachable from the internet.
@stop-treachery that's the plan! I'll have it so public instance operators can contact me via email or Matrix with proof that they operate a certain instance, so I can give them API credentials that will identify them and prevent one person from depleting the pool.
@sclerometer it won't prevent private instances, they'll just have to pay for their own proxies or use free ones. Why should I finance private instances?
@zedeus is there any instance in public that works now or there isn't any way that we can have nitter instances like before 😭
@zedeus possible for private instances to just manually provide account credentials so they don’t need the guest keys? My instance is purely used to grab rss for some bots and I think it would fly under the radar so if that’s an option seems like a good solution for lower volume people.
For my Use Case i make a few requests to the nitter instance in public to just get the latest tweet that's it is there any way i can do it
With guest accounts, will +18 content finally return to nitter?
I think it's possible because both guest account and real account only need to provide oauth token and oauth token secret in the client, but it's much more troublesome than using cookies
Maybe, it can load tweets those media flagged as sensitive
@dawnerd it would be easier for you to use automatic guest account(s), there's just a weird inconsistent limit to how many you can get per IP per day, hence the need for proxies for bigger instances. If your instance has 1 user and you don't expect to make more than 500 requests every 15 minutes, then you won't even need proxies.
@zedeus do you know when this will be added to nitter.net?
@zedeus Will you at least tell us where we can find some good proxies when this is implemented?
What the hell is a proxy, can someone explain this in simple terms? Will nitter be up again or not?
Some people call proxies VPN, proxies on easy words are servers which will make requests to some domain you want to request from their side and redirect output to you
From what I understand, in the context of Nitter, a proxy would just be a service that takes a request meant for Twitter and routes it through a different server, which has a different IP. That way, one server hosting a Nitter instance can use any IP address out of the list of available proxies for accessing Twitter, instead of only being able to use its own single IP address. The point of this is to bypass blocks that Twitter has on the number of accounts that a single IP can use.
I think it's likely that it will be up again, considering Twitter's current architecture and the supported guest accounts, but it's possible that it will take a while for zedeus to develop the proxy service and for Nitter instance hosts to reach out properly. The official instance, nitter.net, will probably be the first to go back up.
In my personal case, nitter is an interoperability interface for noscript/basic (x)html browsers. I was interested only in the anonymous access to read the twits of a few people/orgs, because they do their announcements there.
As far as I understand and roughly speaking, we now need a guest account per IP (which is non-sense in an IPv6 world, unless having an accurate tree of worldwide IPv6 prefixes).
Then in my case I would need some fixed IPv4/IPv6 address to create such a guest account probably with a yt-dlp kind of tool, and use that same tool to list the twits from some twitter accounts.
And nitter as a generic noscript/basic (x)html service would want a pool of such guest accounts/IPs (expecting nitter users to be reasonable and to deplete not the caps of guest accounts too quickly).
Don't forget, this is big tech: They want to force you to use their web engines ([blink|geeko]:google,[safari]:apple) and nothing else. Look at what is doing "kick.com": you don't have the "right" web engine (even browser), you cannot even access the site, and more and more big tech web apps (they aren't web sites) are doing the same, enabled by the "tech" of some shaddy companies.
They will force you to run all-the-time changing browser finger printing javascript programs in order to generate all-the-time changing auth credentials during a user session.
Ofc, they know nobody (unless Big Tech itself) will be able to implement a real alternative web engine (for instance coded in plain and simple C) able to handle those permanent javascript challenges.
In my personal case, interoperability tools like nitter can do so much without being official. Digital freedom needs regulation... hardcore big tech regulation for interoperability (for instance for push protocols we have irc, etc).
Well, I think nitter can give its community a try, but bad actors (whatever the reasons) may kill it for good.
@zedeus Your desire to not support private instance access to your proxies makes sense to me.
For small private instance users (like myself), will the code to create the automatic guest accounts be included in the nitter codebase, or alternately as a standalone thing?
Aside from donating money to pay for proxies, do you think it will be possible for others to help in this effort by running a proxy and working with you to add it to the pool? This seems like it would be a much more clearly scoped commitment than running a public instance of nitter, in that (I'm guessing) traffic and use would be fairly consistent and predictable.
Perhaps instead of a set of proxies it would be easier to request the guest accounts via Tor. That should use a random exit node for each request, yet you would only need to use one proxy on the client side (the address+port of your local Tor daemon). The only problem with that is if too many people use Tor for this, then the exit nodes may gradually hit the daily limit and it wouldn't work all the time. Or if Twitter blocks all known Tor exit nodes outright, that would also be a show stopper.
So... it works. I'm going to implement it properly and get it live on nitter.net ASAP, then work on the guest account API.
@TheHCJ working on it, later today probably.
@net-tech sure, I can include a recommendation in the readme.
@natmey yes. I'll include it in Nitter one way or another, as a fallback if you don't have access to my API.
@WOnder93 that would likely be too slow and unreliable, if it even works at all. The proxy service I use has millions of residential proxies ("ethically sourced" whatever that means), and I doubt I'd get the same with Tor, but I suppose it could be an option.
@gfpj while theoretically possible I don't think that would happen. Besides I'm not creating accounts, I'm not agreeing to any ToS. They're "guest accounts" that get automatically created when you open the Android app for anonymous access. Don't give them good ideas though 😛
nitter.net is officially back, with everything working better than ever before. Search is even back to functioning 100%, no more weird limited results requiring refreshes. Tomorrow I'll work on creating a simple service to acquire and provide guest accounts to public instances, and a simple fallback for small private ones that don't want to deal with proxies. As for now, please use nitter.net to your heart's delight and let's see if it remains stable.
For those confused about 403s on nitter.net /rss URLs, see https://github.com/zedeus/nitter/issues/937
I'd just like to thank zedeus and everyone else working on nitter. Not only have they built some quality software, but also continue to overcome twitter's restrictions in clever ways. Your ingenuity is inspiring! Thank you!
Not all instances are working though. I use unofficialbird.com and it still doesn't work
Only nitter.net is working, this isn't in master yet nor is there an established path forward for instance operators to get guest accounts yet.
Search still requires refreshes for me on nitter.net
thank you very much, @zedeus.
@ParamoreFan I'm not seeing that behavior at all. Do you have an example query that makes it happen consistently?
That's not nitter.net
I used Ublock Origin to collapse some elements, so the nitter text on the top left is gone
Sorry, I don't mean to derail the thread but are rss feeds going to be back ? On nitter.net they seem to be disabled.
https://nitter.net/NASA/rss
RSS feeds are only disabled on the main instance, nitter.net. zedeus has said that they are planned to be back in the future (https://github.com/zedeus/nitter/issues/937). Once other public instances can acquire guest accounts, you'll also be able to use any of them for RSS feeds.
@ParamoreFan The verified icons are also gone. Are you sure this is actually nitter.net? Is something weird going on with your browser? I can't replicate it no matter how many times I refresh.
I'm not ParamoreFan but I was able to replicate it occasionally:
Around 30 minutes ago, I refreshed around 30 times and only saw it happen once. Around three minutes ago, I saw it happen around 30% of the time for around 50 refreshes. Now, I have refreshed around 15 times without seeing it. So, it seems to be inconsistent.
I have Brave Shields on in the screenshot, but I've been able to replicate it with them on and off, so I don't think they're affecting it.
oh how I tired me have I of elon musk, I can't even enter my twitter account now x.com.. thanks zedeus now it works very well nitter as you fixed it.... nitter.net works in all old browsers with opera 8 with TLS 1.2 on my nokia s40, on android all the browsers where I tried it, the app. The truth is that without your invention of nitter I would not have been able to view twitter I couldn't even see my inaccessible twitter account... thank you very much!! :D
@michaelskyba the rate limit for search is only 50, so it's possible but somewhat unlikely that it's already getting hit on all of my 1500 guest accounts. I'm using the same search endpoint and thus small rate limit for user search, but I'm going to replace that tomorrow with another one.
I used ublock origin to get rid of verified
I saw it myself, looking into it.
First: Many thanks. It started out well, but now I get: "Instance has been rate limited."
First: Many thanks. It started out well, but now I get: "Instance has been rate limited."
I was swapping out accounts and doing some code changes, it's working again for now.
Yes! And thanks again!
@zedeus Nitter.net has not worked for me at all for a few weeks (other instances have, subject to the Twitter/X restrictions). Depending on the browser and configuration, I get a blank page with either "This site can’t be reached nitter.net unexpectedly closed the connection" or "This page isn’t working nitter.net didn’t send any data" messages. Does nitter.net block any IP addresses?
nitter.net blocks over 68k IP ranges owned by hosting companies and businesses, and hundreds of abusive IP addresses. If you want me to check whether your IP address is blocked feel free to send it via email.
Hmm. Worked well for some time. Now I get again: "rate limited" or sometimes "no item found".
@zedeus thank you so much!! Two q: is it possible the tweets & replies is broken? I can see timelines when it's just tweets, but it's 'No items found' if I click t&r.
Are gifs supposed to be broken? I'm just super grateful it's back up and running please don't get me wrong, I just don't know if it's my addons or nitter having some issues due to EM breaking everything (regular users were btching on twt some hrs ago about its video feature)
Same here
It (nitter.net) worked last night but give or take 20% of the requests ended up with "No items found" where the list of tweets is supposed to be (everything else like profile banner, sideinfo etc incl background picture at profile header seems to be working all the time). Fixed by reloading the page 1-4 times.
Now 100% of the requests ends up with "No items found" at the tweetlist :-(
Seems like the IP addresses used by Mullvad are blocked too cause I can't connect to your instance.
Not related to this issue but may helpful for some guys:
For myself I solved the problem with this Twitter to ActivityPub (e.g. Mastodon) bridge: https://bird.makeup
Oddly, it doesn't seem to be working for me at all now, it just keeps loading indefinitely.
same for me to
when will the issue will solve?
For me, the page load fines, but I only see this message, regardless of refreshing. https://nitter.net/nearcyan works, though, but not https://nitter.net/nearcyan/with_replies.
Now tweetlist seems to be working (for the moment) :-)
Or rather the /username works but not /username/with_replies (who still gives "No items found").
Swapped guest accounts again. Using 37k accounts instead of 800-1500 this time, should help prevent or at the very least delay these rate limits.
Seems to have resolved the previous issue with /username/with_replies not working.
Now both /username and /username/with_replies are fully functional - thanks! :-)
The main Nitter instance must be very busy, being the only functionnal one at this time. I prefer to wait a few more days that the others runs well again than switching to the main.
Any idea when the other main public instances will be able to work again ?
great work.
please consider unblocking mullvad ip's like already mentioned here.
thanks.
So, blocking VPNs by nitter.net is intentional or coolateral?
Collateral, they tend to use random data centers. Mullvad has a list of IPs but I can't find a list of IP ranges, and I'd rather not throw 700 IPs into an iptables file.
understandable.
i will try to optain the ip ranges.
could take 2 days. will report back.
Please do not scrape nitter.net
of course not. i asked their support to contribute. they have a good attitude towards inquiries.
RSS feeds are cached for 15 minutes, so it makes a big difference in how traffic affects the service.
Multiple accounts at once not quite working?
https://nitter.net/AccountableGOP,RonFilipkowski
No items found.
https://nitter.net/AccountableGOP,RonFilipkowski/with_replies
works. Sort of. Not all images/integrated web links show like they do on privacydev.net instance.
Interesting, I'll look into it
I'm just happy that the main instance is working again. Thanks for all the work you put in @zedeus. I have my own instance running on the Docker image, but I will happily wait until the image has been updated.
Hi !
Every user I try on my nitter instance returns user not found. I seem to have a different error from other people ?
All I’m trying to do is getting the latest tweets :)
Non-official instances are not expected to be working because the development of the guest account service hasn't been finished yet.
Ok thanks !
@BANKA2017 when running the onboarding/task.json end-point to get the guest account I'm actually hitting the following:
Ever seen it go to here before? Looks as though I'm ending up with OpenWelcomeFlowLink instead OpenAccount being returned.
why is the rss path returning forbidden? https://nitter.net/search/rss?f=tweets&q=test&since=&until=&near=
@ImTheDeveloper Yes, there are situations where the response does not contain OpenAccount, and I don’t know the reason for this yet.
@ImTheDeveloper They use a flow with various steps to handle account creation, guest account creation, logging in, etc. There's a step for verifying your email for example. The app navigates these steps, but I haven't found a way to cancel that flow to return it tho the beginning, except for using a new IP address. If someone could figure out a way to reset the flow that would be wonderful.
@Thorvarium it's disabled still. It'll be turned back on when all of this is over and things are stable enough.
All Mullvad IPs have been added to a whitelist.
thanks. that was quick. unfortunately nitter.net still times out.
@zer0fl0w can you shoot me the IP address you're using in an email or on Matrix? I'll look into it
Thank you for your amazing work.
So far the improvement is huge after the blocking event.
I wonder why the Nitter.net instance is retrieving 16 minutes of posts in the search result timeline.
After that the bottom message i've got is "no more..." and there is a lot more tweets after that. The search query is "nasa" in this case.
I don't know if that limited search result is related to an ip problem or the amount of request being fetched on the instance.
@zedeus done (mail).
Is there some aggressive throttling going on at nitter.net right now?
All of a sudden I got this after 5 windows reloaded:
Edit: I assume some glitch in the matrix since its working again.
Thats a feature I have completely missed.
Where can I read more about how its supposed to be working along with are there any upper limits (except for URI length) for how many accounts can be squeezed into the same request?
Also since it currently doesnt seem to be working - is it expected to return?
@zedeus Is it possible to unblock TOR?
@Apachez- yes, very aggressive throttling. Did you hit it during normal browsing or are you scraping? If so, please don't.
@orbea it wouldn't be easy and I'd rather not due to the amount of abuse by scrapers. There are many other instances available (at least there will be when things go back to normal), including onion links.
Thanks for the info - I've had one actual open account in the last 50 attempts 😂 but at least proven it worked. I'll run through proxy and see if it helps.
I've been doing some digging around GitHub and there's a fair bit of usage of the tasks to automate blocking users and such but the particular OpenWelcomeFlowLink which looks to be a deeplink in android isn't mentioned anywhere. Typical!
Is there an ETA on when/if these fixing changes are going to be pushed to master branch?
@ImTheDeveloper Indeed! I was ripping my hair out playing around with it. I understand how the whole open_link stuff works and how flow navigation is supposed to work, there just seems to be no way to reset this flow. I even tried the logout.json endpoint to no avail.
@TempUser13 Pushing the changes to master isn't enough, instances with more than a couple of users need several guest accounts and large instances will need hundreds if not thousands, which requires some infrastructure so Nitter can acquire them automatically from a service I'm going to run. I have no ETC on this but I'm working on all of it.
It may be worth to push these changes for those of us who are running a private instance?
Would it be possible for an update that let's you use a non guest account for those who wish to self host until you've built out this service for getting guest accounts?
@rderensy they're already pushed to another branch, see #985
@TempUser13 see #897
Thanks again for making Nitter work again. But i got this bug where sometimes a tweet/reply has link in form of
https://nitter.net/<user>/status/<id>/photo/1. When i clicked it it just redirects back to the tweet/reply.@Dheatly23 that's intended, not a bug. Twitter does the same thing, doesn't it? What behavior are you expecting?
If it's the same behavior as I have seen a few times it appears that the tweet contents (usually an embedded image/video) are replaced with the link,and since the link itself just leads back to that same tweet that content is not accessible.
I initially dismissed it as representing an unimplemented feature ala spaces but going back they seem to just be normal tweets.
Every time I post a Twitter tweet, it appears to me that it is not possible to view the tweets of this protected user. Please, what is the solution to this problem?
@Fatmaana protected user? It's impossible to visit those tweets without that user following you, so Nitter could never support that.
هذا التطبيق Twidere for Twitter/Mastodon
@Fatmaana Twidere is unrelated to Nitter. I suggest you ask on their GitHub repo if you're having problems with it. This thread is for discussing a specific issue with Nitter.
How do I communicate with them
Their website says to use their GitHub issues for support.
This really isn't the place to ask about unrelated applications.
(@zedeus This conversation has been resolved, so you can go ahead and mark the exchange as off-topic if you want, to avoid it cluttering the thread. 🙂)
I noticing old 1.1 search endpoint is working with OAuth although it's not working anymore with guest token.
x-rate-limit-limitheader on search endpoint is 180, much less than graphql endpoint is 500.Thanks @yekoc for confirming my suspicion. I first suspect it was something with GIFs, since i never saw it ever since this update. It looks like this (link) :
Any ETA for RSS to return for user profiles on nitter.net?
I checked this API few hours ago and it was working, but now it is not. Is this downtime temporary or permanent?
zedeus has already said in this thread that RSS will be re-enabled after the situation becomes stable. That will only happen after the development of the guest account service is finished, and zedeus has said (also in this thread) that there is no ETA on that, but that they are working on it. Therefore, there is no ETA on RSS feeds returning.
(Edit: Never mind, sorry: I shouldn't have swung)
Temporary, and it will recover in a few days.
You can try to change the ip address and do again.
I hit it during normal browsing but perhaps Im doing this inefficient?
Currently I have each user I wish to read their tweets in their own tab (approx 20 tabs in total). I have removed nitter from startup pages in my browser and instead created a html page with javascript that opens up a new tab every 0.5 second (the delay can be altered to make it go even slower between loading each tab).
However when I later refresh tabs I go through them one by one (ctrl+tab) and hit F5 and thats when I hit the throttling the other day.
Would using a multiuser url be more efficient and less demanding on the nitter side of this?
That is instead of having 20 tabs I should do this instead (which would be a single http(s) request towards nitter)?
https://nitter.net/user1,user2,...,user19,user20/with_replies
Scraping when I sit in front of the browser and read the tweets?
Scraping is when you have a bot doing stuff for you which I dont.
That script is because chromium, chrome and firefox lacks capability of having a "seconds between tab to open on startup" when you use "open these urls on startup".
So I resolved that by using "open about:blank on startup" and then having a html page on my desktop which I doubleclick.
That html-page contains a javascript that with a delay will open the tabs I wish (the twitter accounts I wish to read through nitter along with forums, news pages and what else).
So without this script having all nitter urls as "open on startup" would trigger the throttling at nitter side within 1 second just when I open the browser (since the browser opens ALL these "open url at startup" at once which from nitter point of view looks like a flooding attempt of 20 concurrent requests at once). Using this script the tabs are opened in a more slow pace which works excellent.
The problem I encountered is that when I some time later want to find out if there have been any new tweets I cycle through these tabs pressing ctrl+tab followed by F5 and thats when I recently triggered that 504 error after reloading the 5th tab.
Didn't realize its this quick to get a 429 self hosted
@Apachez- It's on purpose stuff like that gets throttled. That isn't normal browsing, you're causing more load than 20 users combined. The multi-user timeline feature uses 1 request per load, so it's extremely efficient compared to opening them all separately. It's apparently broken right now but you can switch to tweets & replies to work around it.
@TempUser13 Are you using one or more guest accounts? What kind of load are you doing? (e.g. requests per second)
@zedeus it is normal browsing since you would normally add the users you want to read their tweets in startup of the browser. That is one user per tab (which is why tabs exists to begin with). At least when you use twitter without an account or account to tweetdeck or similar aggregator.
Do there exist some limit of how many users the multiuser supports other than that the url must be shorter than 2048 bytes or how long an url can be nowadays (seems like general recommendation is below 2048 bytes per url while Google Chrome supports 2 megabyte (2097152 bytes) url)?
Edit: I tested the multiuser-feature and it worked like a charm in my case and with the amount of accounts I want to read tweets from - dunno how I have missed that feature for this long :-)
The feature generates a search string that looks similar to "from:user1 OR from:user2" etc. and Twitter limits search quries to 500 characters. If you get above that amount Nitter will tell you, but it's around ~25 accounts per multi-timeline.
Using a non guest account on #897 trying to refresh 323 RSS feeds
I've seen there is an "end_flow" input but it doesn't seem to react at all either.
Also for account creation I can't workout whether there is some fundamental delay in generation of the oauth guest accounts requiring a second call to the open_link to get an account open or whether it's the rotation of IP that does it but you can go through patches of accounts opening right away and sometimes requiring 3+ calls for it to happen. For your mass creation did you just fire off a bunch all at once and take what worked or did you keep retrying? Ive been manually opening up accounts with postman to try and figure out how the flow.json endpoint works as I'll need to do this every month it seems.
A new release of docker container (zedeus/nitter) is scheduled?
@stopmotio nitter.net might be, can't deal with all that traffic. It's not bandwidth, it's now needing more and more guest accounts to keep up with scrapers.
@ImTheDeveloper I have a simple node.js program that gets a guest token, flow token, and then tries to start the welcome flow to get the guest account made. Each step is retried once if it fails, so I've just been running it with 10k proxies, about 100-500 proxies concurrently. I've been able to make 190k accounts so far. I could've made hundreds of thousands already but I haven't smoothed out the process yet, still takes a bit of manual work.
@guillaumekoto Yes, right now the tests are failing because there's no guest account for the test suite, but I'll figure that out asap.
@TempUser13 That's way too much for one account to handle. You might be able to do 323 main tweet feeds every 15 minutes, as the rate limit should be 500. If you're getting tweets & replies, media or search feeds, then it's not possible with just one account.
@BANKA2017 great work on this, question so I can learn, how do you reverse engineer the android endpoints? do I need to use something like android studio or what do I use? thanks!
@DrSocket I didn't reverse engineer the Android endpoint, I only used Fiddler to capture packets, consumer key and consumer secret are searched from Google.
I have received a feed from this instance this morning.
..(https://nitter.moomoo.me). I am no expert, I'm just using the service, what and how did this happen? If all instances are not functioning.
It was for an account with 200K followers. (Gov official account)
Huh, https://nitter.moomoo.me/elonmusk/with_replies and the search tab loaded for me once with tweets there, very strange.
Ran some authentication tests and it's just acting weird again where sometimes an endpoint is accessible with a given bearer token (e.g. the iPad one) and sometimes it isn't. This isn't something we can work with unfortunately, but maybe they will roll back the restrictions at some point.
I am really not sure if it was working the days before, because I have an android app with 15 minutes refresh rate for the feeds (over 30 for me) , and I am checking it every day. This just happened today. and it just sent another feed few minutes ago.
(With replies)
I don’t use the main Nitter instance, and my bots seems to work again since this afternoon.
Envoyé à partir de Outlook pour iOShttps://aka.ms/o0ukef
De : mahthori @.>
Envoyé : Monday, August 21, 2023 5:16:13 PM
À : zedeus/nitter @.>
Cc : edel79 @.>; Manual @.>
Objet : Re: [zedeus/nitter] looks like X/twitter(?) broke something again (Issue #983)
I am really not sure if it was working the days before, because I have an android app with 15 minutes refresh rate for the feeds (over 30 for me) , and I am checking it every day. This just happened today. and it just sent another feed few minutes ago.
(With replies)
—
Reply to this email directly, view it on GitHubhttps://github.com/zedeus/nitter/issues/983#issuecomment-1686526218, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALELY26ANLTRCVOCMXD3XKTXWN3T3ANCNFSM6AAAAAA3QWLU6U.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
I don't understand what you mean, can you explain more and if there is a
solution to this problem
في الأحد، ٢٠ أغسطس ٢٠٢٣ ٣:٢٥ م Zed @.***> كتب:
What is the actual rate limit for userTweetsAndReplies? I assume it's a lot lower than the 500/15m quoted earlier for basic tweets but trying to figure out exactly how many guest accounts I should aim for to make things stable if I'm polling this regularly using my local instance. (Up to 15 right now though it's kind of hard to know when I have enough since a lot of the older ones have already been limited and won't be usable again for a while.)
I'm not sure, I haven't done enough testing to test what it really is. As low as 1 request per second is enough to trigger the 24 hour rate limit, but I don't know if it's window-based. I do know that once you hit this strange rate limit, it immediately affects the endpoints for tweets & replies, media, search, and individual tweets (tweetDetail). That is, if you exceed the rate limit for media, you are now also unable to access search, media and individual tweets/threads for 24 hours. The main endpoint used for the main tweets tab is not affected and can't hit the same rate limit from my extensive testing.
In my own testing it seemed like with 5 accounts polling every 30 seconds it was able to keep going for a few hours but got limited eventually. Raised it to 60 seconds for now at least until all the other accounts come off the 24 hour limit again. (All on tweets/replies on a single user profile for a notification thing I'm running.)
Can someone confirm if guest account creation is still working. I am also getting this same response as @BANKA2017
Still working here. Seems that happens sometimes though. Guessing if the ip is already used?
Sometimes it appears there is a delay in the creation of the account so you need to wait a few seconds/minutes and then it will create if you call the next_link again.
As mentioned it can also be an IP restriction issue, so rotating IP will cause the account to create. For all the fun, it can be a combination of both!
@OrangeFlavored Any update on your testing?
For anyone wondering how to login with a username\password, this snippet below should be helpful.
I'm using a requests session in Python with the headers based on https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357.
After task1, you pass the "att" response header as a request header to task2.
Username is sent in task2.
Password in task3.
Task4 should give you the final auth. It gave it to me on the first try, but after running the code again, it asked for a confirmation code that was sent to my email, which is sent in task5. If correct code, auth is returned.
@0o120 Thank you very much for your snippet! I was using @BANKA2017 node code which works great too. Do you have any snippet in Python to use the oauth_token/secret instead of the guest token?
I was doing it this way using requests-oauthlib
Just passing it as the auth kwarg in requests.get()
Not sure why, but this specific user doesn't work.
https://nitter.net/Min81937515
There's still no instance supporting RSS, right?
It seemeed like it had been going well for a while, but after many hours eventually the guest accounts started to get limited and now all are as far as I can tell... Was going with like 10 guest accounts~ and a 60 second interval since some of the ones I created weren't usable at the start.
Nitter html is easy to parse, I am parsing it using residential proxies and it works well. Sometimes I have to retry feeds several times or switch proxies. But it's good enough for my use case.
@emv33 what instance are you scraping? Please don't scrape any public ones, especially not now that guest accounts are required which easily get rate limited.
@terr72 see here https://status.d420.de/
I've been using the 'guest_accounts' branch for a couple days now, using just 3 accounts (could probably get away with just one) for RSS feeds needed for IRC bots. I only use one feed with replies as it seems that will hit the rate limit fast. All direct timeline feeds have been working just fine.
So if you want personal RSS just clone that branch, grab a guest_token using the node.js script mentioned in this thread and save it to guest_accounts.json in base directory.
I be able to create a third party x client using this guest branch and using a users credentials instead of a guest account to read and write xeets?
Hi
I’m currently installing the guest account branch. Could you please tell me how to create guest accounts and how to set them up with my instance ?
Thanks !
@zoubidabida The script posted in https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357 can be used to get guest accounts, though you'd only be able to get one using your own IP. I edited it slightly to proxy the request so I could get more:
Then just copy the returned objects into guest_accounts.json (you may have to modify them to add quotes around the key names since the ones I got back didn't have them) in an array and they should be loaded by nitter.
It's a bit of a hassle finding free proxies that work and haven't been used already for the same purpose, though I managed to get a decent handful of guest accounts with some patience. Obviously you could automate the process more to make it easier though I haven't really bothered yet.
@OrangeFlavored change the last line to
console.log(JSON.stringify(account));@OrangeFlavored thanks !
Where do I put the json file once I have it ?
@zoubidabida It should be saved as "guest_accounts.json" in the main nitter folder. You can see an example here of how it would look with 2 accounts though obviously I've redacted the actual credentials. Also I think many of these fields may not be necessary here but it's not like it does much harm leaving them in...
awesome thanks !!
I'm getting this error :
reject(new AxiosError( ^ AxiosError: Request failed with status code 400 at settle (file:///home/nicolas/twaccounts/node_modules/axios/lib/core/settle.js:19:12) at IncomingMessage.handleStreamEnd (file:///home/nicolas/twaccounts/node_modules/axios/lib/adapters/http.js:570:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { code: 'ERR_BAD_REQUEST', config: { transitional: { silentJSONParsing: true, forcedJSONParsing: true, clarifyTimeoutError: false },Probably an issue with the proxy you used. Make sure it supports SSL, and in general if you're using free ones a lot of them just won't work so you may want to try a few.
If you just want one guest account you may be better off just using the initial version of the script for now (which will just use your own IP) which will get you one account without needing to find a proxy.
Ok I switched to the no proxy version and I'm getting "undefined" output.
I'm sorry to bother you with this, I'm not used to nodejs :)
Most likely your IP has already been used to get a guest account then, I think this process is used for the Android app so if you're already using that maybe it already used yours up though I don't really know much about how that all works.
that's strange since I'm using a private ip.
I guess I need to wait for zed to implement a general solution then :)
It worked with a separate computer !
Thanks !!
@zedeus Hi, I applied the guest_account branch and I'm getting this error :
fatal.nim(54) sysFatal Error: unhandled exception: field 'elems' is not accessible for type 'JsonNodeObj' using 'kind = JObject' [FieldDefect]I don't know how to solve this.
Thanks :)
You need to put the object in a list, like so:
[{"user": ...}]I'm so sorry, of course.
It works, my nitter instance is back up !!
thanks
I used to make around 40 requests every 2 or 3 minutes. It was each time a get for a user. How many requests per hour do you think I can make now with the guest accounts more or less ?
Looks like nitter is up and running but whats the deal of rate limit? How many requests can we put in to not get rate limit and scrape smoothly, would love to get any idea if someone has been testing it out
It seems like from my experience and some others here checking a user's basic timeline can be done fairly frequently (I saw the number 500 requests every 15 minutes posted earlier though I don't know if it still holds), but checking tweets+replies is much more strictly limited, and even rotating through 15 accounts and checking once per minute would eventually lead to getting limited. I haven't really tried other endpoints since they're not relevant to my needs. For now I've decided to switch over to doing both, checking basic tweets every 20 seconds and tweets and replies every 3 minutes, and see if that works to prevent anything from getting limited.
Alright , thanks for the info. I'll test some out myself and see how it works
for me this just returns
{'errors': [{'code': 215, 'message': 'Bad Authentication data.'}]}on task 1. double checked the account logs in so it's not a problem of username or pwd.@DrSocket have you checked that you are using all the correct headers including guest token from: https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357
I created a simple shell script today that is another alternative method of grabbing guest tokens. I have one that uses proxy servers but it sifted though over 1000 today and those that did connect were already denied guest tokens. For this I tethered my phone to my computer and just cycled the data on/off to grab a new IP.... Settling on 10 for now.
https://gist.github.com/cmj/6e6f6cae51c28cf6e161ceba8d108dda
Edit: @AlyoshaVasilieva Thanks for the catch.
The shell script has some syntax errors, for example
-H 'X-Twitter-API-Version': 5, 'X-Twitter-Client: TwitterAndroid'should be-H 'X-Twitter-API-Version: 5' -H 'X-Twitter-Client: TwitterAndroid'My crappy attempt at writing tooling for getting guest accounts, since I have been runing a personal instance to try to alleviate load on the public ones.
It is notably different from the other examples by:
The stub frontend the absolute bare minimum required for me to get accounts for my personal instance. The backend package can be used for more elaborate things. But if I'm being honest with myself "it's nothing special, and very Not-Invented-Here, because I dislike Node.js".
https://gitlab.com/yawning/twitter-guest-account
Just an update for anyone who was interested that things seem to be relatively stable with this method (checking basic profile tweets every 20 seconds and tweets+replies every 3 minutes) and I haven't seen many accounts getting limited since then.
hi ! So you check one account every 20 secs with a simple get /user ?
Yeah, all on the same account, scraping http://localhost:[port]/[username] or http://localhost:[port]/[username]/with_replies if it's been more than 3 minutes since the last tweet+replies check.
Ok I’m using much more. I do 46 users every 5 minutes (separately). Yours is 180 an hour and I’m doing 552 an hour without any problems :)
Hey does anyone know what it means if I'm getting a response from the guest account flow which doesn't contain an open_account key with the necessary tokens. There are a lot of subtasks such as sign up but nothing useful by the looks of it.
@dylanp-git it means you can't do it again from that IP for another 24 hours
Windows users... here is a Power Shell implementation to grab guest tokens quickly.
Tether your phone to your computer, and run this as many times as you can. Disconnect and repeat.
Output:
user : @{id=1694805356575068160; id_str=1694805356575068160; name=Open App User; screen_name=_LO_0824v6z12eN; user_type=Soft}
next_link : @{link_type=subtask; link_id=next_link; subtask_id=OpenAppFlowStartAccountSetupOpenLink}
oauth_token : 1694805356575068160-HC5qZBoTAcyt9jTu8U0ZLRBbJTBXXX
oauth_token_secret : 1GT1jMrcNh5gkVDCAj7n8rxsl6CIkEvioxemNfSKc2XXX
attribution_event : signup
will there be a mode to make nitter grab the guest tokens directly as it needs them? applying for guest tokens in a central place seems annoying, running separate scripts as well.
my instance is very low-traffic (to the point where I am reasonably confident I do not need to even switch IP), and even if, I can already point nitter to a
SOCKSHTTP proxy to facilitate that.@untitaker That seems like an incredibly complex undertaking. I made 272 calls per hour with 3 tokens, I now have 20 only because I wanted figure out the process to grab them. With 3 guest accounts I didn't see any limit errors.
I would expect this not to be some backend tool that grabs more than one token beyond your working ipaddr. I had a proxy script run on 2000 IP's and only 3 returned with guest tokens today; many were already taken.
If it's private, or a small instance you can probably just get away with one.
@untitaker you can use socks4 too https://i.imgur.com/2vLMawn.png
Run this on an Android mobile in Termux, switch to flight mode and back between runs and you get a stream of tokens in no time. Don't over do it so others also get a chance (although it would not surprise me if Xitter were less restrictive with mobile IP ranges when it comes to guest token generation).
Hey, sorry for the late reply, had to work on something else. I got it running, it's all good. thanks a lot for the script! the 'enter_text' logic is in case it asks for email verification code right?
Yea that's correct. 1 time it asked me for the confirmation code sent to my email, then a separate time it just asked for my phone number connected to the account. Both times I entered it, it worked.
Would anyone be kind enough to share an example (python preferability) using just the oauth token and secret (guest tokens) in a Twitter API call? Like querying the latest ElonMusk tweet? Tweepy and other libraries require the consumer key and secret. Any help would be greatly appreciated.
@Allen-Taylor You need all 4, consumer key and secret plus oauth token and secret, otherwise you can't sign a request.
First, thank you for all you do. Nitter is amazing.
Banka's example uses:
const TW_CONSUMER_KEY = '3nVuSoBZnx6U4vzUxf5w'
const TW_CONSUMER_SECRET = 'Bcs59EFbbsdF6Sl9Ng71smgStWEGwXXKSjYvPVt7qys'
These are used to create the Bearer token.
Are these using HIS consumer account creds or are these general consumer creds for the Android guest account?
If I am supposed to use other consumer creds, where do I get them?
Thank you for taking the time to answer my questions. I'm learning a lot from this thread.
Those are tokens used by the official Android app, so don't worry about it.
Have you been able to generate any tokens today? Tried some of the scripts that worked but no go...
Yes, I don't have any issues generating tokens.
I am struggling to recreate the extended header successfully. I am constantly getting the error 401 on my side when trying to make a request.
Can someone share how the header should look like for a successful request?
It should be like this
Authorization: OAuth realm="http://api.twitter.com/", oauth_version="1.0", oauth_token="1692750343510335488-RDoNrQYUxlNe9A5wygl1JZMJs4oS3F", oauth_nonce="169250878645316925087864531692508786453", oauth_timestamp="1692508786", oauth_signature="fykucfKBx7da6qwzE4AQtwupEA4%3D", oauth_consumer_key="3nVuSoBZnx6U4vzUxf5w", oauth_signature_method="HMAC-SHA1"Also you might need to check your signature, your signature might not be valid.
I would like to ask directly in order not to miss the subject. Can we access tweets with python as in the nitter documentation or do we need to use another method?
I successfully updated my self-hosted instance to the guest_accounts branch and a guest account generated from my own IP.
Everything works well except suddenly user search returns nothing. Searching tweets and directly viewing a user's tweets are all fine, and log shows nothing. I think it is not due to the guest account's rate limit because I can still search for tweets. Did I miss anything?
@jiatern what commit? Check the about page at /about
@zedeus
The version shown in the about page is 2023.06.05-1234b9b, butgit show HEADshows that I am on the latest commit on the guest_accounts branch.Nevermind, I started from scratch and rebuilt nitter, it now shows the correct commit 2023.08.26-7630f57. All is well now!
Hi guys, it's amazing what you've managed to develop by discovering the new X endpoints. Let me ask you something, for searches through 1.1/search/universal.json I read in your previous posts that the rate limit is 50 per day per account, if so, to achieve 800 searches a day, would I have to get at least 16 accounts and rotate them?
My admiration to @zedeus , @BANKA2017 and to all those who have done their bit to get us to this point.
@Fa5g It's not 50 per day per account. It's 50 per 15 minutes per account.
And that rate limit is for the graphql search.
I don't know the rate limit of the 1.1/search/universal.json request. I don't think nitter is currently using this type of request anyway.
First of all, thanks zedeus for Nitter and everyone else for research or contributing in one way or another.
Second of all, here's a Python implementation for grabbing guest accounts/tokens, based on Allen-Taylor's Powershell implementation:
Hi guys, thanks Zedeus and this community for the amazing job you are doing on this project. I'm using Nitter to scrape tweets from the search and retrieve the location of the users. Right now, I'm using the oauth_token and oauth_token_secret in the headers to make the requests, and Express VPN . But lately, I've been getting blocked a lot, and changing the tokens has had no effect. If anyone can give me advice or has experience with something similar, I would really appreciate the help.
For anything beyond light scraping (a few requests per minute at most), you're going to need a lot of guest accounts. Twitter has never been known to hand out IP bans besides rate limits, so I wouldn't worry about using a VPN.
I’m currently scraping 46 users every minute with 10 guest accounts. I use GET /user 46 times and I haven’t been throttled since I started that a week ago
For /user it's less of a problem. If you do /user/with_replies, /user/media, /user/search or /search then you need a lot more guest accounts.
As I myself learned not too long ago nitter supports multiaccounts to make the load less on the nitter backend.
That is instead of:
You can instead do:
The above also works with /with_replies such as:
Are the 10 guest accounts each using a different IP address?
Yeah I used separate IPs when I created them.
Do you also rotate your IPs when you scrape data (GET /user)?
No I do everything from my own instance. That’s why I use the many guest accounts
You can instead do:
httpx://nitter.net/user1,user2,user3
Not working wihout "/with_replies" : get the "no items found" message (with existing accounts, of course).
De : Apachez @.>
Envoyé : mercredi 30 août 2023 06:59
À : zedeus/nitter @.>
Cc : edel79 @.>; Manual @.>
Objet : Re: [zedeus/nitter] looks like X/twitter(?) broke something again (Issue #983)
I’m currently scraping 46 users every minute with 10 guest accounts. I use GET /user 46 times and I haven’t been throttled since I started that a week ago
As I myself learned not too long ago nitter supports multiaccounts to make the load less on the nitter backend.
That is instead of:
httpx://nitter.net/user1
httpx://nitter.net/user1
httpx://nitter.net/user1
You can instead do:
httpx://nitter.net/user1,user2,user3
The above also works with /with_replies such as:
httpx://nitter.net/user1,user2,user3/with_replies
—
Reply to this email directly, view it on GitHubhttps://github.com/zedeus/nitter/issues/983#issuecomment-1698494056, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALELY26M4WLIKQRFGYXSUZDXX3CDBANCNFSM6AAAAAA3QWLU6U.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
What would the structure of a guest account be like? It would be like a header with the oauth_token and oauth_token_secret
Thanks for all your work in this regard @zedeus and beyond. I was drawn back here as I get "No items found" on my nitter instance.
I can never seem to get the docker build working so rely on the docker image on Docker Hub. Any signs when a new updated image featuring a guest account system might be available?
Also for general people - are we any closer to working out rate limits for function (user/search), per IP? Is it possible to create like 50 guest accounts on the same IP or will Twitter eventually just rate limit the IP instead of the guest account?
--
The first two curls work but the last gives an empty output?
Here's a modified version of the script which automates the harvesting of tokens when run on an Android device through termux and bash on a rooted device (needed to control airplane mode) with WiFi disabled. It will attempt to get a token, toggle airplane mode, wait for a random period of time and repeat the process. New tokens are appended to a destination file.
To use the resulting file you need to add square brackets to the beginning and end to enclose the list of objects into an array. Either edit the file itself or run something along the lines of the following (where tokens is to be replaced with the file containing harvested tokens):
Here's how to use the script, remember it needs to be run on a rooted Android device (tested with Android 9) with an active data connection and wifi disabled since it uses the fact that those devices get new IP addresses on every reconnect:
...and here is the script:
Might be wise to create lots of real accounts. Since browsing Twitter on the app without being logged in was removed, who knows how long until they just nuke all the guest accounts and disable making them. In my opinion, a handful of real accounts would be the way to go to be better prepared for whatever comes next.
Agree I don't see this lasting long..the surge in guest accounts is a huge give away now unfortunately
Yes, it is doubtful things will remain working for long this way, they'll eventually start tying guest accounts to some identifier - IMEI, IMSI, whatever - to limit the number of guest accounts useable from a single device or do away with them altogether. As to making real accounts, well... maybe. They'll be able to put a limit on those as well by tying them to phone numbers or - $deity forbid - some form of ID document.
not sure if it's related to this issue? but I'm getting This site can’t be reached. I can't even access https://nitter.net/about
@notherenorthere you probably got IP banned. What are you using nitter.net for?
@notherenorthere @zedeus Same thing here but nothing out of the ordinary, just eventual browsing. ISP got banned perhaps?
@rbfowler9lfc I was setting up a new rate limiting strategy that seems to have had some false positives. This should be fixed now.
oh?? just browsing! after I posted this I figured I should've first checked on google if it was down and when it said it was down I'd check from time to time if it was working okay. Because it didn't go out completely, it went out, then it was normal again and then it died again and a few mins ago I checked and it was still down and says the same here https://downforeveryoneorjustme.com/nitter.net and here https://updownradar.com/status/nitter.net however I'm not sure how trust worthy those sites are
it's working now please ignoreeee!! I just checked hehe
Edit: dead again lol ahhh I'll check it out later I'm a bit busy rn :(((((
Working again here, thanks @zedeus !Edit: Checked now and it's down again on home internet but working on cell network, around 2023-08-31T21:55:00Z
@notherenorthere @rbfowler9lfc Please email me your IP addresses that are blocked and I'll take a look to figure out why.
I emailed you @ the addy in here: https://github.com/zedeus/nitter I don't copy+paste it in case we can't do that
I think I just found out why it happened, should be fixed now!
@zedeus I am getting below error:
syncio.nim(855) readFile
Error: unhandled exception: cannot open: ./guest_accounts.json [IOError]
What could be the reason for it. I am running my own nitter instance locally with the guest_accounts branch
The guest_accounts branch expects a guest_accounts.json file (placed next to nitter.conf). See https://github.com/zedeus/nitter/pull/985#issuecomment-1685963751 for how it's formatted
Any update for the docker image zedeus/nitter?
@zedeus There is still censorship that affects nitter.net.
https://nitter.net/i/status/1697308269528654180#m
@guillaumekoto not yet.
@orbea nothing I can do, depends on where the instance is hosted.
@zedeus no longer banned here, thanks again!
Happend here too, mail sent.
Using multiuser requests as in https://nitter.net/user1,user2,user3/with_replies
@Apachez- thank you! It helped me identify an overlooked edge case, which should now be fixed.
@zedeus Thanks! :-)
@the-blank-x I am also getting the same error. Can't seem to resolve it. Can you help?
I have placed guest_accounts.json at the same level as nitter.conf.
Format of guest_accounts.json:
Steps to run:
The docker container cannot see the file, add
-v $(pwd)/guest_accounts.json:/src/guest_accounts.jsonafter-v $(pwd)/nitter.conf:/src/nitter.conf(or before, the order doesn't matter too much)Does anyone else get the same output running the various "create guest account" scripts in this thread?
How come my output doesn't have the
.open_accountkey?@RunlevelConsulting This happens when the guest account has already been created in a previous execution on the same day from the same location (IP address).
-or-
This happens when this is executed from a location (IP address) blocked from Twitter/X. Maybe you are using a VPN? Try to disable it before executing the guest account creation.
@the-blank-x Its because of the formatting error in guest_accounts.json file. This happens when you copy paste. Please try creating an empty file and enter the details manually first to test it. I had the same error and was able to fix it.
@zedeus Is there any reason to ban IPs on nitter.net ? Every Mullvad IPs seems to be banned.
@sajiljosephs: In their case, it's because the container cannot see guest_accounts.conf (it says
Error: unhandled exception: cannot open: ./guest_accounts.json [IOError]). In the case of a malformed file, it'd sayError: unhandled exception: input(1, 4) Error: { expected [JsonParsingError]@obito: See https://github.com/zedeus/nitter/issues/983#issuecomment-1684987665
@obito
This is because people abuse VPN and proxy solutions to scrape tweets.
The community invites you to setup your own instance to scrape content
Did anyone get an updated docker image to work with the guestaccounts branch and guest_accounts.json file and mind sharing theirs? I keep getting
IGSEGV: Illegal storage access. (Attempt to read from nil?)in the container logs. I see the files in the container so not sure what exactly it can't read. Here's what I did just in case my process wasn't right:docker build -t guestimage:latestdocker compose up -dto start nitter and redis togetherOn main branch it was at least starting fine but not able to load any data from twitter so i figured it was a guest accounts issue.
I got mine to work without any issues by pretty much the exact same process than you.
Did you make sure
guest_accounts.jsonis mounted to the right place inside the container? It should be in/srcfolder, same place than yournitter.confthanks for confirming @ojaha065 and pointing me in the right direction.
Just in case anyone else runs into that error: I was missing brackets around my guest accounts json object and that's the error that manifested in docker. (it was not an array and couldn't be parsed)
hi, I have another way make it working, fetch data via browser by use socks5 proxy.
The full steps you can see here twitter spider example
This way can keep csrf token always fresh.
tip: a chrome browser page need 360M memory in my production envirmoment
I'm really don't believe this is the case, I ran the following script once only, on a newly spun up Ubuntu server in AWS and still get the "OpenWelcomeFlowLink" response. I see a few others suffering the same issue. The Twitter API definitely doesn't block AWS IPs (or at least it didn't before they started charging ridiculous sums for the API).
I also ran this script on my home computer which is just a regular internet connection, no reason for it to be blocked.
Can others running the guest_account creation script give details on the environment they're running it on?
Edit: Should I make a separate issue for this? Also, I presume a guest account creation script has to be run to generate credentials for the guest_accounts.json file on the guest_accounts branch, correct? There's currently no automated way to create accounts in the branch?
Edit2: @ImTheDeveloper is right. I ran the script through a few proxies, doesn't always work but the success rate is high enough to generate accounts. Cheers!
I'm running my own script rotating proxies through brightdata. Very rarely the first request will provide the oauth tokens, normally it takes 3 or 4 attempts. I usually give up after 4 attempts and move to a new flow token, that said I've removed this logic in my script for a few hours now and the time taken to return the open_account (by requesting multiple times with the same flow token) appears to be completely random. Therefore, the only variable is which proxy is used.
Don't forget, the guest account setup on twitter side has to be async also. They are expecting a real user to take some time to follow the flow steps. We could simply be hitting some processing delays on their side too.
Edit: I've done some testing of 6 proxy services. You'll feel like you're mining for bitcoin trying to use pools of ISP or dedicated DC IP addresses. You really need to go for residential IP's and even better mobile residential. I've switched to mobile IP's and getting a token every single request.
Ps https://techcrunch.com/2023/09/08/x-updates-its-terms-to-ban-crawling-and-scraping/
List of scripts found in this thread:
Original Node.js https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357
Node.js with proxy https://github.com/zedeus/nitter/issues/983#issuecomment-1689430618
Shell Script https://gist.github.com/cmj/6e6f6cae51c28cf6e161ceba8d108dda
Go https://gitlab.com/yawning/twitter-guest-account
PowerShell https://github.com/zedeus/nitter/issues/983#issuecomment-1692369656
Python https://github.com/zedeus/nitter/issues/983#issuecomment-1697288218 & https://github.com/zedeus/nitter/issues/983#issuecomment-1715487727
Android Termux https://github.com/zedeus/nitter/issues/983#issuecomment-1701267067
I can't generate any more guest accounts. Is it just me? It appears the URL isn't working.
Working for me
Yeah it's been working ok for me too, keep in mind that the 24 hour cooldown seems to reset every time you attempt to generate a guest account, so if you keep trying every few hours you're never going to be able to make a new one. I've just been using my own IP and a script to get a new account automatically and add it into my guest_accounts.json (also removing any >30 days old, though I'm not sure what the exact limit is) every 24 hours and 5 minutes just to be safe. So far it's all been working well.
Can you share your script?
Sure, here:
You may want to set PRUNE_INTERVAL to None initially until you've generated a decent pool of new accounts, because it's going to remove any of your old ones that lack the creation_time attribute. (Or you could just add it manually to roughly the date/time you created them if you know it.) I run this once per hour through cron, and it uses the LAST_ATTEMPT_FILE to track when it should next attempt to make an account.
@OrangeFlavored
Many thanks for your script and everyone else who has shared one.
There is just one thing I'm uncertain about. If new tokens are added to guest_accounts.json does Nittter need to be restarted?
Yes, I have it scheduled to restart every day at 5AM for that reason.
I've tried using BrightData residential and mobile proxies, but without any success. I haven't been able to set up a single account. I use #983 (comment) script and change the proxy with
'http://brd-customer-**_***-zone-mobile:*****@brd.superproxy.io:22225'@MaxosI Since it takes 4 requests to create a guest account, you probably have to keep using the same IP in multiple requests.
There is an explanation on the BrightData site how to achieve this.
From my testing it seems to take a few seconds to be IP banned and running the script in a simple loop is netting me a number of tokens. Last run was 10 and a couple before that was 5.
I have some questions about the guest accounts:
@jcmag (speaking as owner of nitter.services.woodland.cafe)
you need 40k accounts for a public instance, but that is after you have banned the majority of bots. there was somebody claiming every guest_account can do 499 requests/day, but in my experience much more than that is needed.
I found that anything above 100k will make nitter consume a lot more memory than necessary, but it will run fine even with 1 mio accounts in my experience for as long as you give it multiple GBs of RAM
no, i don't think so. zedeus said in Matrix chat (linked in README) he is planning to fix this as next step, but i am not sure it happened
guest accounts last for 1 month
generally, if you have any sort of efficient way to acquire multiple guest_accounts a day, it tends to be the case that you can easily generate 2 mio accounts per day, meaning that you can just generate the entire guest_accounts.json from scratch every day. this is wasteful but solves some of the questions above
keep in mind that overall hosting a public nitter instance got a lot harder, and it's not just because there's no docker image or the need for guest-accounts. the biggest timesink after that is abuse prevention. Speaking as an instance owner i feel that scraping of nitter instances and general abuse got a lot worse over the past 2 months. See https://github.com/zedeus/nitter/issues/1015#issuecomment-1714063128 as well.
if you need further help setting up guest_accounts specifically, please hop onto Matrix (linked in README). right now I don't think the core team has a lot of bandwidth to provide support for earlier setup steps though.
As someone just running a private instance for some light scraping, I find 15 accounts is fine for one /user request every 20 seconds and one /user/with_replies every 2 minutes. The with_replies is the main bottleneck for me since it seems to get accounts rate limited really easily and I could probably do with less accounts if I didn't bother with replies, though the script I posted above should be fine to maintain a rotating pool of 30 or so accounts without effort on my end anyway so I'm not too worried about it for now.
Also as far as the "how are these accounts managed by the code? if an account doesn't work, does nitter try the request with another one? (and how many retries?)" I believe this was updated recently and requests will be retried with the next available account. Requests are not retried at all, as far as I can tell, and sometimes things that appear as connection errors (at least from what they say in the logs) are also treated as an account having been rate limited.
What about user/media? I have some media-only feeds and have been wondering if it would help to convert them to user and just set up a filter on my end. Although I'm also worried about losing tweets due to some limit.
@OrangeFlavored @zipfile6209 /user is practically unlimited, aside from the 500 request per 15 minutes per guest account limit. You will run into the strict rate limit doing /search (and of course /user/search), /user/with_replies, /user/media, and /i/status/...
Requests do get retried once with a different account if it fails due to a rate limit, but that account may also fail, so there's no guarantee it will succeed. I'll probably turn this into a config option as smaller instances have more leeway when it comes to increasing max retry attempts etc.
@untitaker
I'll look into the memory usage. A lot of data is stored, but it definitely shouldn't need that much RAM, and I suspect most of the usage is from the initial loading stage since memory doesn't get released afterwards. I'll switch to a faster JSON library and probably require JSONL instead of JSON to parse it line by line, also making scripts that acquire accounts a bit simpler.
It did: https://github.com/zedeus/nitter/pull/985/commits/fcd74e8048362fcf8284871ee067099e8de28a89
When an account expires after 30 days, the next request made with it will fail with a specific error code. Nitter detects it and removes it from the in-memory pool, but the file will not be updated since it's a temporary solution.
Does anyone have a working example Python code for getting the latest Tweets from a user with created guest account credentials (but without using Nitter)? What headers/payload do I have to send to which API/endpoint? Does the request have to be from the same IP that was used to create the guest account?
All the endpoints and features are located in https://github.com/zedeus/nitter/blob/master/src/consts.nim
Headers: https://github.com/zedeus/nitter/blob/master/src/apiutils.nim
Whats next?
https://x.com/thesource/status/1704294353693270085?s=46&t=5qaU4aptMKPZtXjSkIh6CA!
Thanks for having created and maintaining despite all the changes Nitter!
I use Nitter on my private instance to read timelines via RSS using feedly. Thanks to the
/.tokensdebug endpoint, I noticed that feedly browses very aggressively each item of a RSS feed when viewed on the website and the app - triggering sometimes over 10 tweetDetail requests in less than 1 second and worse, not caching these results. Viewing / refreshing a busy timeline can very quickly consume 100's of tweetDetail requests. These requests all originates from35.187.132.0/24and are very likely used for summary / screenshots renderings within the feedly app.Denying access on the reverse proxy for
35.187.132.0/24solves the problem. Without this rule, my ~15 guest accounts of the private instance may all get limited within 20 min of continuous feedly browsing. I guess that aggressively caching tweets (typically when the timeline is rendered for the RSS feed) could also solve the issue. Hope this trick may help other feedly users!if you're using a rss reader and you're concerned about how hard it hammers your nitter instance, consider doing https://github.com/zedeus/nitter/wiki/Rate-Limiting-%E2%80%90-Nginx
might be temporary? hoping it's not elawn doing elawn things but..
502 Bad Gateway
nginx/1.22.1
Temporary downtime, messed up the server
List of nitter instances:
https://status.d420.de/
This one works https://nitter.poast.org/elonmusk/status/1704747213736292828#m
Twiiter needs to understand this will NOT make people form accounts, it will make them drop Twitter. Today I had to tell some folks I cover in my online news reporting that I would need direct links or better yet video files over Signal or I would be unable to post a report on today's activities.
I neither have nor want corporate social media accounts. I already block all of Meta and have for a decade, plus Google is relegated to Tor-only access, I find Twitter's chronological-only timelines for non-logged in users 100% useless, so I have entirely stopped using their service for anythng but extracting videos or photos from linked posts. Some of the folks I work with have STOPPED POSTING TO TWITTER due to the combination of harassment of non-logged in users and all the reinstated hate accounts.
If Nitter goes down and stays down, Twitter will join Meta on my list of useless servers
I tried this script on an OVH server, for the first few days it wasn't picking up any token, I had to generate some beforehand on my home internet connection.
But when I took a look at the guest_accounts.json, it started generating token on the OVH IP, so I guess you have to just keep going until Twitter thinks your IP is safe?
In my experience, you can only request a guest account once every 24 hours. Any more requests just pushes your request window out another 24 hours. I have a
@dailycronjob that acquires a guest account just fine.I've created a small webservice that allows owners of public instances (i.e. listed in the wiki) to fetch 50k guest accounts every month. redacted, it's dead
let me know if you have any feedback, by sending me an email. Do not put more noise into this issue.
@drpepper66 Great service. Is everyone sharing the 50k guest accounts ?
no, distinct. it is intended to cover the needs of any large public instance.
@drpepper66 Cool, could you share some technical details behind it?
@tonywangcn all i can say is that it doesn't do anything special that isn't already mentioned in this thread. the service around the script is also boring.
i see you have written blogposts about web crawling before. right now i want to ensure that normal human usage of nitter can resume. i can share more at a later point.
@drpepper66 about how much later?
Did anyone else notice a big change in how fast an account gets rate limited?
@DrSocket no, not really at all. the same tricks still work since multiple weeks now
Can someone give me a TL:DR on how to fix this? I am a bit over my head
Has something changed with the guest account process? I've noticed within the past 3 days or so I've been unable to get any new ones with the same process that had been working up until then. (Using new IPs rather than just waiting the 24+ hours on my home IP didn't seem to make a difference either.)
Still working for me. I'm using your script modified to run in a loop and I got 20 tokens on Saturday and 9 yesterday. It is very variable how many I can get a day. I run it on my own IP.
yes, a lot of IP addresses no longer work. it seems they went on a banning spree, possibly ASN-based. it is also possible that certain people became too greedy and polluted the pool for everybody, but i think it is unlikely.
@drpepper66 When did you notice this? I'm still able to do
@dailycron jobs. I also run against a few proxy lists and "phone tether script" once a week; haven't noticed any change out of the ordinary.My daily request randomly gets denied and I'm going to assume it's some time precision issue. My last denied request was today. One could bump to every other day...
1-2 days ago. Exact timeframe unclear. Most IP pools I have tried no longer work reliably. If you have proxy lists that work for large volume of accounts (10k+ / day), please email me them or any hints you have (link in profile). I do not recommend posting publicly. The above linked service for public instances currently does not work reliably.
The user search endpoint seems to be down. is it working for anyone? on X the url still seems to be the same
user: /search?q=xyz&src=typed_query&f=user
latest: /search?q=xyz&src=typed_query&f=live
Seems to be working now.
@lukefromdc https://nitter.net/search?f=users&q=elon this works for you?
https://nitter.net/search?q=%23test
works,
a user search does not
that's what I said...
I do NOT know enough about website programming to understand much about this, so letting that go. Nitter is something I use, but my programming experience is limited to DE and UI type stuff
Something borked a few moments ago with https://nitter.net:
"
Instance has been rate limited.
Use another instance or try again later.
"
It seems the guest accounts are exceeding the rate limit too fast. Was the rate limit changed?
I was experimenting with using fewer guest accounts, nothing else changed. It's working fine now with a larger batch.
What is the rate limit, requests per guest account in a 15 minutes window?
Depends on the endpoint and requests per second, it's very weird. A lot more guest accounts are needed than expected based on the 15 minute rate limits.
Invidious released smart-ipv6-rotator to circumvent YouTube blocking (docs). Haven't been following the issue here closely but am curious if it's relevant as a solution.
Hi, no info on User search ?
On Sat, Oct 14, 2023 at 18:00, Zed @.***(mailto:On Sat, Oct 14, 2023 at 18:00, Zed < wrote:
The legacy endpoint used stopped working, I'll replace it soon
Slightly off-topic but since this is the general complain thread I would also like to add some positive experience.
Since about yesterday it feels like nitter.net loads much faster, dunno if thats a coincident or not (about when those rate limited messages went away)?
That is both for a specific username such as:
https://nitter.net/< user1 >/with_replies
But also the multiuser loading:
https://nitter.net/< user1 >,< user2 >,< user3 >/with_replies
I was having OK success but noticed the last couple of days I'm able to get anywhere between 600 - 2k tokens but then hit upon a couple of limits:
The above can be both fixed by simply killing my processing and waiting an hour or so. Maybe there is a wider rate limit on this now.
I have 40k-50k twitter/x new tokens on a daily basis. If anyone required feel free to contact me at raka01001@gmail.com
twitterminator is also back up and running. we're able to generate 100k tokens per day, which should be enough for the instances we are currently serving. please email me if you need something for your private instance, I can approve those manually but may have to kick them out if twitter implements further restrictions and resources become tight again.
I would like to point out that there's likely a threshold of traffic at which the twitter team notices how hot their servers run, becomes active and implements further restrictions to save their servers. please do not hammer the API at high concurrency and produce spiky load on twitter's end if you can avoid it -- I believe it is better to load data at a constant, but low request/s. I myself have been guilty of this.
I have noticed the same behavior on myself when fighting bots on my own instance: I don't care about bots that send 1 request every 10 seconds 24/7, but if you are trying to scrape a batch of data in a very short amount of time, every day, and come at me from 1000 different IPs to achieve 200 requests/s, you are getting banned, because I do not want to pay for those additional resources.
Just be polite.
https://nitter.net/Support/status/1714429406192582896#m
Interesting move if rolled out everywhere
Any payment requirement (even a penny) to make a new account serves as backdoor identity verification and limits account formation to folks already familiar with using money online,which many are NOT.
If THAT happens, new Twitter account formation should nearly stop. Many folks (myself included) who do not shop online and thus do not have any online currency accounts and/or do not do real names social media will respond to this demand by closing the tab and walking away from Twitter. Meanwhile, no such hassles over at Mastodon. Until Twitter rolls over and sinks for good, we need Nitter. When it does, I just hope Mastodon and not Meta is the main beneficiary
I usually create one daily guest account from 4 different IPs (one of them at a University and another in a VPS). For several days I have been unable to create any account in any of those IPs -- have they started banning home IPs too? 🤪
I generated a few just now using 5 different IPs (4 mobile, 1 residential) and the yield was pretty low but still enough for my private instance.
May I know the process on how you are generating guest account tokens ?
Yeah, in the end just in case I waited 48hs. instead of the usual 24hs. and I could get some accounts too, great.
There are both python and javascript scripts along this thread, make sure to "Load more" items in order to find them.
Hi, is there a way to subscribes to a couple of account using the guest token? If not, is there a way to poll the endpoint? And whats the rate limit like?
I'm trying to run my personal instance using the latest commit (
d175832) but it says it's rate-limited despite having dozen or so guest accounts.All errors have this structure:
error: RangeDefect, msg: value out of range: 3462608142 notin -2147483648 .. 2147483647, accountId: (...), url:(...)@cherryband Can you post one of the account that triggers this error?
@zedeus none of the accounts have succeeded, and I'm not sure how nitter chooses it, but the first one is
{"oauth_token":"1721163358626271232-XJNegxOgOBaSjMQPJAT6ZEb5hGAO3O","oauth_token_secret":"juTU729TzjsC9GvmrTN5m41TE9dPMIcEuOlTFcynpPLx6"}ok trying it on my desktop it works fine, does this software not support 32-bit environment?
It should, but I might be missing an explicit int64 somewhere
I had some fun diving into all the comments on this issue yesterday when I wanted to get Nitter running on NixOS. I documented the process for getting the
guest_accountsbranch working withservices.nitterso it should be pretty easily reproducible for people running a NixOS server.Thanks to the Laughing Man and everyone whose hard work makes a very important piece of software like Nitter possible. 🙏
Hello all,
I had a free afternoon today, so I gave a shot towards getting myself a private instance of Nitter, that I had used for some time until the API changes hit. I have a few other Docker containers running such as for Reddit and YouTube. Unfortunately, I have to give up on this for now as I wasn't quite able to figure it out...
This is in no way slander, I understand this is all very active development. However, I didn't want to leave without at least saying thank you all for such amazing work.
If there is ever anyone willing to create a tutorial for how to build a docker image for this guest_account branch from scratch, I would be extremely grateful. Even more so, if the changes get merged into the master Docker image, that would be beautiful! Until then, I will keep an eye on how this continues to develop!
Thanks all!
@GensHaze This is not a comprehensive guide by any means, but here's something:
Prerequisites: Have Git and Docker installed
1. Clone the repository, switch to
guest_accountsbranch2. Use Docker to build the image
Tag can be pretty much anything.
3. Update the docker-compose file
Change
image: zedeus/nitter:latest-->image: nitter:guestnitter:guestshould be the tag used in step 2.Create a JSON file and mount it by adding
to the volumes section around line 11.
./guest_accounts.jsonshould be path to the JSON file you created. For simplicity's sake I just keep it in the same folder than my docker-compose file and nitter.conf.4. Get some guest accounts
This is the hardest part. You can get one quest account per 24h per IP and each account is valid for 30 days only. (So you need to keep generating new accounts manually or even better, automate it somehow.) There's scripts for generating accounts in multiple languages in this thread. For a single-user instance only couple of accounts should be needed, depending on usage ofc. I use around 15 and never have had any issues.
Add the accounts to
guest_accounts.jsonas follows:5. Start the containers as you normally would
e.g.
docker compose up -dTo add to @ojaha065's guide, someone also pointed me towards this service for fetching guest accounts using
curl.(@drpepper66 Really nice authentication mechanism, by the way! 🤩)
https://twitterminator.x86-64-unknown-linux-gnu.zip/
@ojaha065 Indeed, I was so close - just a few pieces I was missing which you just now handed to me, thanks so much, I truly appreciate it a ton!
After finagling some more with some Docker networking things, I am now running my own private Nitter instance fully functional with 10 guest accounts.
I will regardless continue to keep an eye out, see where the future takes us 😄 hope Twitter doesn't keep making things difficult, well... one can only hope!
Here's a PHP version (yes I know, boo hiss) based off a Python script that was posted a while ago. It's not the cleanest but it does the job.
It does the following:
This can be run automatically, and in theory, keep adding new accounts to your guest_accounts.json file. You just need to restart your Nitter instance every now and then to load in those new accounts.
Very cool, but how to add/integrate this php to my guest_account instance?
Not sure I understand your question, you don't integrate it into the guest account instance. It runs on the host (usually with Nitter running in a Docker container).
Presumably you have PHP running on your host, so you'd just put that code into a script and run something like
php myscript.php. The script writes to/tmp/guest_accounts.jsonbut that can be changed to anything.When you launch the Nitter Docker instance using the guest_accounts branch, you should have the guest_accounts.json file mounted as a volume which the instance then uses to validate API calls.
If I'm misunderstanding, please elaborate on your query a little more so I can help.
@RunlevelConsulting Thanks. Thanks to your help, I was able to recover the nitter I use on my home server.
I have an idea for how to save Nitter and actually make it pretty much indestructible. And not only Nitter. Any kind of privacy-friendly proxy. And on the top of that, reduce the burden of running an instance and shortage of instances for some proxies, like ruraldictionary for example.
It draws inspiration from onion routing/P2P. In it's simplest form, people would launch a localhost server and receive requests from other nodes. Circuits will regenerate with each request for better privacy.
Let's say that this software will register a route for any supported backend and listens on port 9001. The backend itself doesn't need to run on our machine, the goal is to maximize the use and usability of such proxies as nitter, but some sort of algorithm would probably be in place to ensure both sufficient anonymity and latency and discourage leeching when someone would only run the most lightweight and least data intensive backends.
So we go to localhost:9001/nitter and we see what we'd normally see on our local Nitter instance. But our instance has not directly connected to twitter. Instead something like routing our nitter instance via torsocks happened but without the whole pain of being tossed into one bag with abusers of the Tor network, rendering our instance unusable. Instead, our request was passed through a couple nodes which might not even need to run nitter unless they're the exit node.
The software would work a bit like DHT, let's say each request includes two headers:
provides- list of services accessible at our nodewants- what service a node must be running to send the response to our requestGreat idea! I can't wait for users to use my bandwidth, and have my IP in Twitter/X's logs in case they end up accessing content that X/Twitter (or the US government, since they've subpoena'd information from the company in the past) deems sensitive, putting me at risk (with either my ISP or law enforcement). Sounds great!
@154pinkchairs cool idea but this is off topic for this issue thread.
@koitsu The sarcastic comment is not welcome here.
@koitsu you can always rent a VPS. There are a couple providers that allow anonymous payments.
There are some (not)PHP script around there? We are limited to modern stack.
@satoshinotdead see the wiki page: https://github.com/zedeus/nitter/wiki/Guest-Account-Branch-Deployment
For anyone else coming here with new issues with acquiring guest accounts with their previously successful scripts (like this one) make sure to update the version of your twitter user-agent to
10.10.0like in example script in the linked wiki page above.Is that both "User-Agent" and "X-Twitter-Client-Version"?
Can you elaborate on why this needs to be changed? I managed to grab number of tokens a few hours ago with my existing script.
In my case I couldn't get any tokens with any of my IPs until I changed both to 10.10.0. If it still works then theres no reason to change it.
Looks like something broke again last night...
tweets, with_replies and media all returns 0 messages currently :-(
Example:
https://nitter.net/knytanders
https://nitter.net/knytanders/with_replies
https://nitter.net/knytanders/media
Edit:
Related: https://github.com/zedeus/nitter/issues/1136
Also seems like twitter itself is actually down so not an Nitter issue for now.
Edit2:
https://downforeveryoneorjustme.com/twitter
https://downdetector.com/status/twitter/
Twitter is down, nothing broke.
Could perhaps nitter inform about that somehow other than just "No items found"?
Like expand that message into "No items found (no reply from servers)" or such to give a hint if the user really do have 0 messages or if there is some other malfunction going on leading to 0 messages are being displayed?
Because from experience past year or so the "No iteams found" means that twitter have changed something in their API again to break Nitter (and others).
I just found experimental direct attempts to reach Twitter hitting their login wall for every user tested (just a few).
I do not have (or want) an account so no idea if the site is responsive behind that
Seems to currently working again for the accounts I have tested, also the multiaccount listing seems to be working.
That is:
https://nitter.net/user1,user2,user3/with_replies
So case closed (for now ;-)
Thank you for your PHP script. But I got a little error. With the square brackets, the file seems to be unreadable by Nitter. I got this error :
As soon as I remove the square brackets, all is working properly.
Heya, I had that error too, it was from when I manually stopped the script while it was still running.
But yes, the only real error that can occur in the script is a JSON formatting error, so my advice to anyone suffering the same issue - copy and paste the contents of your guest_accounts.json file into a JSON parser, see where things went wonky (probably towards th end of the file), fix it and you should be good to go.
looks like its down again :/
Nitter seems to be down across multiple instances. I dunno why. https://nitter.cz/ is working however, and additionally, the Nitter Instance Uptime page shows that there are a bunch of other working instances.
The domain has been suspended. See https://github.com/zedeus/nitter/issues/1150
If that's due to a complaint by Twitter it might be necessary to create some Torified instances that only talk to Twitter via tor. Slow and Tor nodes can be blocked too but more hassle for Twitter given blocking all Tor nodes would cut off many of their users in countries that ban and block Twitter.
So many people have dumped Twitter and closed their accounts that I am now unable to communicate with some activist groups that migrated to Instagram. I have blocked Meta for over a decade and won't change that for anything. Twitter may end up on that list too the way they are going. If not accessed by 3ed party backends, then no accessed at all and treated as permanently down.
Also, Musk could choose not to "operate in" such countries which for this purpose means not having an office in, having a server in, or selling ads though a local subsidiary inside the offending country. In some cases (such as China or the US) it also means not travelling to the offending country or in extreme cases its allies. Then you have impunity and can tell the offending regime to address all complaints to /dev/null.
I've been unable to play most videos for the last week. The issue seems to be resolved for now. Anyone know what happened?
@magamadov6925 #1147
i have no idea how to solve it. i checked all this post but cant find solution. i installed last verion with docker compose
You have to use the guest_accounts Branche.
And here we go again :-(
Started about an hour ago where a refresh I assume ended up at another loadbalanced node (that worked) but now all requests (even with several minutes between each request) ends up with "instance has been rate limited".
On 1/25/24 18:10, Apachez wrote:
hmm, everything works fine with my instance. maybe try another instance
if what you're using doesn't work?
EDIT: disregard that, nothing works
still happening while doing a search or clicking on a post @ least on nitter.net atm. some other instances do work though
Starting today I'm no longer able to generate guest accounts using the bash script posted in the Wiki. I've tried multiple IP addresses that have not been used to generate an account in the last 24 hrs. Anyone else having the same issue? Might be related to instances being rate-limited.
Yep same issue here
On 25 Jan 2024, 16:16 +0000, Jani Haiko @.***>, wrote:
Same here
Confirming same issue. I'm going through a proxy and it's a different residential IP each request.
Looks like they removed the OpenAccount task which contained the oauth tokens
Currently, all instances are showing the rate limited error, and i tested a dozen of them. Could it be a change from the X/Twitter side ?
Yeah the only subtask now is "Signup" (in various forms). Guest accounts have been fully removed. Confirmed by installing the Twitter apk. Old accounts seem to still work on my instances, so it might just be allowing them access until they expire.
Current json output
Expected ouptut
Same here. Earlier today it was just nitter.net but I guess some codechange that were commited today is behind this so when the mirrors caught up on updates Im getting the same limited error from the others now.
In my case I use /with_replies with infinite scrolling in case that have some connection to this behaviour?
I also use one multiaccount tab (https://nitter.net/user1,user2,user3/with_replies) and 5 singleaccount tabs. I try to have a few seconds between the reloads of each tab and usually dont reload more than a few times each hour.
Same. They removed OpenAccount prop
here we go again ...
My instance https://nitter.freedit.eu/ works well. And it's behind cloudflare. What's important, cloudflare reports three waves of ddos recently. Maybe other instance suffered ddos too.
A temporary rate limit could cause a poorly written DDOS script to conclude the site is down and quit.
In fact, almost all instances are affected. https://status.d420.de/
It's working again.
Some instances will work dependent on if they hit a guest token set that is still valid. Once the 30 days is up for those tokens the instance will be broken again. The main issue is the token generation is dead as per https://github.com/zedeus/nitter/issues/983#issuecomment-1910848654 so without a method to generate such tokens we are stuck.
It seems like https://nitter.net/username/with_replies and https://nitter.net/username/status are broken while https://nitter.net/username works without issues for now (at least it loads without error and seems to have up2date content).
Multiuser seems to be broken aswell (https://nitter.net/user1,user2,user3) with and without /with_replies and /status.
If it is no longer possible to create guest accounts, does this mean that there is now only one way - using regular accounts?
Did they disable the ability to browse tweets anonymously altogether?
Is there a tutorial on how to do so?
Correct
received a DDoS from Hetzner and AWS nodes last friday, and around 2 days ago token fetching started to break for twitterminator as well. I suspect that there was a gradual rollout of the guest-accounts blockade, and it made scrapers more desperate and move to public nitter instances :(
I have received mixed reports of existing guest accounts being invalidated as well vs not, but it's not clear to me whether that was an artifact of the DDoS causing a lot of rate limits or a genuine move by twitter.
twitter apk didn't change recently from what i can tell either. i think the "view anonymously" feature in the app is genuinely broken, but if somebody is still able to use twitter for android without an account, on any device, then that would be really good to know.
if somebody still wants some guest accounts to recover from DDoS please email me (see profile), I still have a lot of guest accounts stashed though they'll all expire in 2-3 weeks
So this is finally the end of Nitter with creating new guest accounts not working anymore?
it's not the end in that you can still checkout nitter using user accounts. the age of hosting nitter for people is probably over, though unfortunately
And how do we use nitter with user accounts?
It's probably over for those with individual instances if your intention is to avoid using X. Because new measures have been implemented to deal with scrapers, lurkers and spammers including the introduction of a $1 yearly fee for new accounts, algorithmic checks to see if you are actively engaging on X and posting regularly. Elon wants eyes on his website from registered users and by no other means because he only makes money from registered users who regularly create content, view the ads and preferably for him, subscribe.
This will also become an issue for archiving websites because they'll face the same issue. Nor can they get the funding to pay the API fees either.
If Nitter dies and Musk makes it impossible to find content on X without an account (e.g no access to chronological timeline), I will treat Twitter as Instagram (which I block) and warn my friends who still post there that I can no longer access their content.
I agree. The main problem is going to be Government agencies and private companies that have replaced their in-house customer service systems with an account on X in order to inform the public of emergencies, urgent alerts or civil information and companies who wish to cut costs and have Elon host most of their infrastructure to communicate with customers regarding service status, enquiries and complaints. What is happening will lock out anyone who refuses to use X from all of that communication with no prospect of any free and privacy focused alternative.
The $1 fee you mentioned has been implemented only in New Zealand and the Philippines so far. Furthermore, new users who opt out of paying that fee can still use Twitter but are restricted to "'read only' actions, such as: read[ing] posts, watch[ing] videos, and follow[ing] accounts," according to Twitter's announcement.
That makes me doubt that users are being monitored for active engagement. Does Twitter have a reason to punish passive users? I don't think so. Passive users still view ads so there's still value for Twitter in keeping them around.
They've already done it before where they've purged passive users from the platform. I know one account of another person personally whose account went quiet and was removed. While passive users may see advertising (if they don't use an adblocker), they don't add to the value of the website of which that content attracts advertising/subscription revenue and he can't also verify that the people lurking are humans and not bots that advertisers desire when purchasing advertising space on X. It's why they've targeted and will continue to target lurkers, scrapers and spammers.
I’ve tried using two real accounts but just get a rate limited message right away so maybe some more work is needed to support real oauth tokens.
hi, just a regular user who browses nitter here, sorry to interrupt. Is there anything we should/shouldn't do to not make things worse or is it irrelevant? nitter.net is fully down, is it a coincidence? other instances are hanging on and some barely there. Should we refrain from using it? how could we help?
Someone is scraping nitter heavily (actually so frequently that being a ddos)
Have you been on Twitter/X lately? Since Elon took over, I have to block a dozen of porn/bitcoin scammer bots per week who like and follow me. Almost never happened before. It feels more like they encourage spammers to sell verified accounts.
But that's OT, sorry ;-)
My only theory, is that a massive amount of bot code out there used guest accounts for scraping, and public Nitter instances were used as a fallback if that failed. Literally every instance reached insane limits the exact same day.
Which led people to believe guest accounts were being removed.
Guest accounts have been removed, they weren't just led to believe that. With real accounts getting rate limited immediately and likely banned, I don't see any path forward for Nitter.
Replacing a rogue front with a rogue APP that connects to Twitter's servers may or may not work, I am no expert in that though.
No path forward for Nitter may mean no path forward for Twitter for me. The mass migration to Instagram suggesta existing 3ed party Instagram sites could use more help. I would prefer to see folks use Telegram and Mastodon, but too much is diverting to Instagram and probably FB as well as Twitter slowly sinks.
Any real info on accounts getting banned? Maybe still possible to have private instances?
You can still use a private instance with accounts you have, I have zero accounts limited; but they will expire of course.
Guest accounts will expire, asking about real accounts getting banned.
Mastodon or other AP protocol software is the answer. Fuck proprietary stuff. Enshittification sucks.
@paulo-roger the thing is mastodon still have lacks of content and engagement compared to X
you can help fix that.
If only instances didn't block each other over the most minor stuff...
Exactly. Mastodon is nice and fun and all until admins realize they can block whoever the hell they want.
Well, it's currently still possible to browse tweets without logging in through the website. The guest browsing through the web uses something called "guest token", which is returned as part of the page content and can be used for some limited API access (seems that only listing tweets of a user and viewing specific tweets is allowed).
Here is a PoC that gets tweet content by ID using this method:
Maybe this could be used to implement at least a partly functioning Nitter?
The guest token has strict rate limits and can only use very few endpoints, not even timeline/list/search...
So Alt-right trolling, illegal content, and spam waves are "minor stuff" to you? God I don't even want to know what you consider major stuff then.
Seriously the people who complain about this shit are almost certainly alt-righties who want "muh freeze peach", if you want something for that, Nostr is where its at.
Ultimately though most people though don't want to join platforms that have constant spam, hate, and flame-wars, so in that regard the platforms and networks without "freeze peach" and have rules and moderation will do much better and have more users and real interaction than the ones that don't and allow people to say whatever they choose without direct repercussions, which will be full of spam, trolls, and people posting threats and hate speech.
You seriously have an Elon derangement syndrome. This here is about finding tech solutions, not about leftist feelings.
Yes. Cope, seethe, and dilate.
guys this is totally off topic, please stop
I agree. It's majorly off topic and extremely disruptive.
Edit: Thanks for helping me take out the trash by presenting yourselves as the chuds you are, I promise you're never getting unblocked no matter how good at code you are.
Does anyone know how normal accounts can be used for nitter? I only check a few users, via some high-reputation accounts, and I'd want to add those
Just thought I'd leave a comment to say thanks for all the work of those who worked on Nitter. No matter what state it ends up in, it as served me very well.
You say you agree, but earlier you maliciously misrepresented another user to accuse him of fascism, and now you threaten and insult everyone who disagrees with you.
I don't know if you are a troll, but please, shut the fuck up and stop derailing such a relevant topic.
It might be possible, though I'd recommend doing it only through your own personal private nitter instance, so the accounts don't get banned due to ratelimits.
I think it depends on how old they are, how often they exceed ratelimits (if ever) and if Twitter's team can or will attempt to investigate manually accounts that they suspect of supplying nitter with information.
I suppose one solution could be to have volunteer twitter users supply data to the Nitter servers from their own sessions via an extension or something like that. It would severely limit the amount of content available but could get around issues of accounts being banned because they would be getting the data from legitimate users and mirroring it elsewhere.
The fact that this post was downvoted so much that had this been 'the orange site' it would have been killed tells it all. Yes, that is the main problem with Twixxer-alternatives, the fact that these tend to become ideological silos where dissent is not only not tolerated but seen as proof of the dissenter being so morally corrupted that any contact with said individual is deemed sinful. Not only do instances in the Fediverse ban other instances for being less than ideologically pure, they ban those instances which refuse to ban banned instances - guilt by association. As long as this practice abounds the Fediverse can not fulfil the role of marketplace of ideas since it actively shields inhabitants of such ban-happy instances of ideas outside the Overton window.
The guest account on the Twitter app (Version 9.95.0-release.0 (29950000)) seems to still work, albeit very rate limited.
I hope there will be somewhat newer workaround for Nitter
probably by using the token that it gets when Twitter/X loads a specific tweet/post?
iirc you can still view tweet/post without logging in, so there should be some kind of token going on there
I agree with this .. I had to create separate accounts on multiple instances just to follow different sets of people as the instance they're on blocked the other instance, it gets annoying real quick
Further writing in collapsible box as it's kinda off-topic
I could understand instance blocking other instance due to different laws in some region, like how many Japanese instances are blocked by instances in other countries because many other countries have different law than Japan that some of the toots Japanese users made which are legal in Japan and many Asian countries are highly illegal in some other countries especially in Europe and America
but some instance block other instance just because of ideological differences, for example:
none of those should be a reason to get blocked on the instance level, cutting off the users on both sides from each other
at most it should just be "silenced", that users on both sides can still interact with each other, but the public posts on one instance don't just appear in the public timeline of the other instance
on Twitter/X you can still interact with everyone regardless what your position on certain issue is with just one account, but on Mastodon (and just Fediverse in general) with the instance admin babysitting their instance, you had to choose which instance to use and possibly hop around different instances many time just to find the one that you can use to interact with most accounts, and then still end up making additional account on other instance(s) to cover the rest of instances that got blocked by your main instance as you still want to interact with people on there in some way or another
Essentially this is why I am letdown that Nitter is gone, because as Twitter is login walling their entire site, people still use it due to it's image of being the "digital town square". But unfortunately there is going to have to be a big culture shift to move people over to the fedi.
Spoilered because off topic
The whole point I was making with this comment too was that simply, the fediverse is never going to ever become a mainstream Twitter replacement for this very reason. Let me make an argument about why this is a bad idea for someone coming over from Twitter as a very active fedi user, without delving into politics like other users talking about this have.On the fediverse, you have to make several accounts to follow people for a very good reason. Let's say you have a friend who signs up for Instance B, and you're on Instance A. Unfortunately, due to people you never interacted with or knew about, instance B has blocked instance A, because a member of that instance made fun of someone on an entirely different instance or made a post that an entirely different instance hated (like say something about cryptocurrency for example), which has now added instance A to say fediseer or something to be blindly imported into their blocklist (at least one website even says that you must import a blocklist to run an instance). Even better, you don't even know your instance is blocked because instance B has made their blocklist visible to logged in users. You'll only find this out when it says follow pending or you don't see your posts on his instance, or even better you can't fetch his account or posts on your instance.
Essentially the only way you can communicate with much of the fediverse is to surprise surprise, make an account on the flagship instance (mastodon.social) and follow whatever their rules are to the T. Or you can make a single user instance, but be warned: if you "be yourself" too much you'll end up on more and more blocklists. If you want to avoid moderation similar to that of Twitter, you can expect to be on blocklists because some guy on your instance had harmful opinions on some topic that gets people very worked up.
As a result of this hindering interactions, many people who are not banned for Twitter for some reason or another end up inevitably going back to Twitter, which is as a result where the big discussions online take place. Elon Musk knows this to the point of calling Twitter a "digital town square", and he can abuse this reputation because thinks nobody is going to go anywhere. Most people quite frankly do not want to deal with any of this to just say something online and will go back to Twitter.com if it means they won't deal with the modern day version of a 2000s forum moderator.
This is exactly why I am disappointed that Nitter is getting blocked by Twitter, and why Nitter was so useful. It was a great way to bypass the login walls of Twitter (which went from "login to view NSFW" to "login to view anything other than a single post in a thread") even as the world continues to use it as a major platform for posting announcements, even as the API became unusable, even as "latest tweets" boxes in sites broke. Twitter has broken the site to force people to login.
Unfortunately as the failed Reddit API protest showed, it's going to take a culture shift and actually moving onto different websites to reduce Twitter's influence, and while journalists and others leaving Twitter for numerous reasons is one thing, they're just going to different walled gardens like Facebook, Instagram (owned by FB), and more. The unfortunate reality is that the only thing that makes people actually leave Twitter is being banned from every big social media site. It'll take something on a massive scale like the Tumblr porn ban to make Twitter irrelevant IMO.
twstalker.com has not worked since we noticed the removal of the guest accounts as well. However, it seems like it's working again. Does someone know about their workaround?
Don't we all love mastodon.art?
I think bird.makeup seems to still be working, my friend boosted a post from it earlier but I think that's on borrowed time to be honest.
bird.makeup was found to be scraping nitter instances for his source code. he was approached by many of us to help him set up his own nitter instance but i don't think he ever did. in which case it's using the list of public instances (failing over to each one) from status.d420.de
Never mind, it seems that it stopped working, replies aren't loading, but tweets from trending and search results do still appear.
sadly we dead over here on the mint.lgbt instance D: will try privacydev's fork
Just be warned, the privacydev pre_guest_accounts tree is only for one account, and you will reach limits fast.
You can't go public on one account at all, use this for local rss, etc.
Didn't know that, used to follow the person who ran it but I unfollowed after some people didn't particularly like him. I can see why now lol
guest accounts updated we're back everyone
If bird.makeup dies because of all this, then a lot of folks who have tweets reposted on Mastodon will have to manually copy them over. Hopefully in my community that can be leveraged into using mastodon as the primary place to publish.
Damn it... Now I wish Twitter would just die already and make everyone - especially artists - use some better platform.
I was getting sick and tired of everyone's art gallery being on Twitter until I found Nitter, which was and still is a godsend, so I hope there will be a new workaround soon.
One thing I noticed as user: The very few instances which at this moment work for me flawlessly all have very strong DDOS protection.
Could this part of a solution? Or are they only buying some time and in few days/weeks they will be gone too?
Public nitter instances were/are being scraped to the point of an accidental DDoS, presumably due to guest account creation being killed. DDoS protection has stopped those instances prevented their guest accounts from ratelimited due to being scraped to oblivion, but it only buys some time since the guest accounts will expire within 30 days.
Yes it is silly. Privacy and free speech is non-partisan. I am a bisexual environmentalist (not that it matters anyway) and I support Nitter and free speech.
What are chuds? Urban Dictionary says unattractive person. If that is the case, I think you will have to ban many respectable programmers. People should be recognised for their merit and skill, rather than their attraction level. If you hate the left that much, then go to a politics forum, instead of GitHub.
Edit: I do not know some of these American terms, so forgive me. I can only go off of things such as Urban Dictionary. My intention was not to insult programmers. Many are attractive. My point is, it does not matter how they look, because their skill is what matters.
Awesome !
what script are you using ? Mine isn't working
Can you share your script with me, please?
What exactly has to be changed? Kindly can you elaborate?
Echoing the above. There doesn't seem to be instructions on how to implement the single-user credentials on PrivacyDev fork.
Could someone help give pointers on how to implement it and where to put those credentials?
I haven't personally used it in a while since I use Firefish and no offense but both the developers of Firefish and bird.makeup seem like they can't be bothered to make any advancement for compatibility. I follow the Firefish developer on GitHub and they've been active here meanwhile things are currently in turmoil on fedi thanks to them not being active there for a month.
My instance admin does have plans to move to Sharkey iirc if the situation doesn't improve with Firefish being abandoned.
this issue has derailed like twice now within the past 24 hours, can we not?
Back in August someone posted a script that gives you the auth token from a logged in account: https://github.com/zedeus/nitter/issues/983#issuecomment-1688495284
If someone can get it working, please share.
Thanks to https://github.com/zedeus/nitter/issues/983#issuecomment-1688495284 and https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357, Twitter RSS is now available on RSSHub again:
github.com/DIYgod/RSSHub@3ae25ddb7aHere you go
What is slick? I am still learning some of these terms.
still using twitterminator, just curl a list from https://twitterminator.x86-64-unknown-linux-gnu.zip
though now i think i ran out of accounts...
found some IPs fetching way too many requests coming from "node" user agents. here's how my blocklist is looking
Will my personal account get rate-limited or banned if I enter my credentials into a personal Nitter instance, and follow another medium-activity account (usually a few posts a day) using their RSS feed in Miniflux to automatically fetch their latest posts?
This is currently the most official & active thread🧵 left to collectively discuss what options remain, if any, going forward. The more off-topic it gets, the sooner it'll get restricted, locked-off🔒, or archived forever...along with the rest of the project. Can we please use what little time we have left to positive brainstorming, only?
Giving thanks 💕 to @zedeus + all contributors & hosters for their amazing dedication all these years — putting in so much work to provide value to so many, for so long — is appropriate (and possibly the only thing left), too. @zedeus has proved himself to be one of the best! 🫡
It is not possible to create new guest accounts, but the ones created previously can still be used.
@stopmotio do you mean user+1@... user+2@...? I guess that's worth a try, but I wouldn't place my bets on it. Things that seem feasible at a small scale tend to break for Nitter. My instance (nitter.net obviously) usually did ~92k API requests per 15 minutes, or 100 per second. At that level you need a lot of accounts to put it lightly (I needed +50k guest accounts to avoid hitting rate limits).
At least when I signed up years ago, Twitter demanded I link a phone number within ~4 hours of using my account.
Any way to create instances of this or something similar run locally by each user (as a "rogue twitter app") without an account and simply accepting rate limits? I would rather be limited to say, one timeline every 5 minutes than zero any day.
What about an algorithm to keep rate limited instances up, notify users that service will be restricted, then prioritizing access from IP addresses not seen in the last hour or so and as Twitter permits data to flow answering additional requests, with users asking for more than say, ten timelines or 100 tweets last in line but serviced if Twitter will alllow it.
I just found that Twitter is refusing on their own site to desplay chronological timelines,
no doubt another attempt to force logins. I'l probably delete my entire text file of stored
Twitter usernames to watch if nitter cannot be saved or replaced.
Did they ever go back to chronological order? cause they had done that last year so that their most liked and rt'd stuff would pop before anything else, very annoying. I stopped checking after a few weeks I assumed it was never gonna return to the chronological order. It's not an issue if you're logged in though.
Kinda off topic but I noticed today on twt it gave error when trying to see /with replies, it suggested refreshing I did.. several times, nothing, So I checked only posts and worked fine, I checked again //with replies and then it worked. Not sure if it was a momentary glitch or a consequence of wtv it is those mfs are doing atm.
For many years, non logged-in users were able to select the order in which tweets were to appear-but only on the desktop version not the mobile version. One of Musk's changes seems to be removing that option on the desktop site too.
Since I do not have an account and never will, no chronological time line when not logged in means Twitter is now useless to me except for reading single tweets to which I am sent or can find a direct link.
For the moment nitter.net can be used in GNOME web but cannot be used from Firefox, as GNOME web will allow you to bypass the certificate warning while Firefox won't, citing a site refusal to accept "insecure" connections and that being defined by FF as including bad certificates. There is probably an option somewhere in FF about:config to force this capability but I don't know where it is.
Assuming this dies with the guest accounts supporting it
Same here. I survive using RSS feeds, not so much for a full reading experience but rather as an alert system so that I know when a site has been updated. Nitter has been invaluable to me in that manner.
In
about:config, try changingnetwork.stricttransportsecurity.preloadlisttofalse.Something sorta locally exists, I'm using Squawker atm but it relies on guest accounts and it will soon die.
Would creating an account using the Google Oauth bypass the captchas?
In an attempt to not get the account banned, wouldn't it be wise to use something like Selenium?
I get that it would get very resource intensive but maybe Twitter is seeing something is off because of the way the Nitter client is interacting with Twitter.
Info here https://superuser.com/questions/1780337/let-firefox-accept-a-website-with-self-signed-certificate tl;dr HSTS needs to be turned off on the nitter.net webserver, that's what's forcing the refusal to accept any insecure connections.
nitter.net is HSTS preloaded so it would need to be removed from that list and then that change would need to be slowly propagated to all browsers' stable release channel. This would most likely take multiple months. There's probably a reason Zed hasn't renewed the certificate (it's easy to do), and while turning off HSTS enforcement in Firefox is absolutely not recommended if you care at all about security, it's that or use another browser to use nitter.net it seems.
For the moment I can get though, thanks to forcing Firefox to allow clicking through the certificate warning in about: config. As a rule I NEVER willingly configure any program to override my orders to it. If you are not logging into a site and not moving private information, a man in the middle attack becomes a much lower risk. Logging into anything with a certifcate error is another matter
But instances like privacydev that use permanent accounts will survive this, right? Especially if they are private.
No one is willing to continue to develop it in this case?
What's a permanent account for nitter? is there some way to use your own account to run a nitter instance?
I ran this and I do get a OAuth token and secret from my account (I see an Android login on my account too), but my instance is rate-limited anyway. Has anyone actually used a real account to get a proper token? I only get intermittent service if I mix the other older guest accounts too.
You can generate an auth token from real regular Twitter accounts in the same manner that the guest_accounts.jsonl uses and operate the Nitter instance that way. But its likely impossible to get enough real twitter accounts to run a decent sized instance. Doing this route is just for private usage for people who want RSS capabilities and a lightweight frontend.
If your twitter account is older that 30 days you need to remove the age check, see here
If you want to see how old your account is, use this simple script:
Your twitter ID is the snowflake portion before the dash in "oauth_token" that's returned, you of course can find your account ID via other methods. Everyone trying to use old accounts need to remove age checks.
You should just remove it regardless if you want to keep using nitter.
New problem: Twitter is now shuffling at least some timelines when viewed over Nitter, same as not logged in on their own site. I think the world is almost done with Twitter.
It's still as popular as ever. Mastodon didn't make a dent in their userbase.
Unfortunately, in my community I have seen a major migration to Instagram (which I BLOCK as it is owned by Meta) as people have largely walked away from or closed their Twitter accounts. I have told people that if they want me to be present at events they must inform me directly as I can no longer see their promos posted to any of the corporate owned sites. Twitter used to be an exception. Nitter bought me another six months or so.
My attempt was a failure, the captcha always restarted for no reason. I ended up creating it with firefox and then trying tor, but even with 2FA it was repeatedly flagged as suspicious. When trying to reactivate it with tor the captcha was so broken I couldn't even try.
The good part is that even after that it didn't ask me for a phone number.
Could you point out some timelines where you've seen this happening? Also, did you notice it happening on a specific public instance, or for all instances?
Tests were on the nitter.net instance and on every timeline checked. If this was
not the case during someone else's test it may have been a one-off. I do NOT
recall which timelines I was checking though
On 2/4/2024 at 11:03 AM, "pin-grid-array" @.***> wrote:
@cmj See #1164, I'll check if this change works for me.
Interesting. I haven't encountered that myself, not on any timelines or any public instances. I really, really hope it's just a one-off, as you said. If chronological timelines disappear, I personally will have zero reason to ever look at Twitter again.
Not sure whether it's of concern but my timeline has completely stopped working in Squawker now even with closing and reopening it
I'm able to view profiles, my following and saved tweets just fine but not the timeline
No errors appear either
@cmj why do you claim that guest accounts can be used past 30d? this has been tested in the past and just wasn't true. do you believe there was a recent change?
@drpepper66 I'm not. This is regarding using an oauth token with a 'real' account, since guest accounts can no longer be acquired. The confusing part is probably using the same
guest_accounts.jsonfile to store the token for now.Yes, guest accounts expired after 30 days. If your real account has an id/created_on date over 30 days ago nitter will not load the account (and remove it if your new account age is over 30-days). So the only way to use older accounts is to remove that 30-day-check. Hope that makes sense.
I've been running it just fine with a 5-month-old throwaway account for internal use. Formatted /.health page:
Total accounts: 1 | Rate limited: 0 | Oldest: Sep 06, 2023 | Newest: Sep 06, 2023 | Avg: Sep 06, 2023There is a python script in one of these threads to quickly obtain the oauth token using account credentials.
Hi, where's the "30 days ago check" to disabled it?
You still have to be using the guest_accounts branch.
Comment out line 205 and build as usual:
github.com/zedeus/nitter@52db03b73a/src/auth.nim (L205)Copy over your oauth_token and oauth_token_secret from your registered account to a fresh guest_accounts.json like before.
Clearly not the best method but works for now until the dust settles.
And again, one of these methods should help you obtain an oauth token for your account
https://github.com/zedeus/nitter/issues/983#issuecomment-1688495284
https://gist.github.com/cmj/c45c46d9dd11fa2d4ba4de55c50598e6
https://gist.github.com/cmj/998f59680e3549e7f181057074eccaa3
@cmj
What is the life of a real account oauth token before the script needs to be run again to obtain a new one?
@thomas-333 That part I'm not entirely sure of. Per docs, they last indefinitely.
But since these are for login sessions and not app permissions, they might be treated differently. Coincidentally, I do have an account with an active session sitting at 87 days, so I'll see if it gets revoked at the 90-day mark (which seems to be the standard for sessions) soon.
One could just create a
@monthlycron job that updates the token to forget about it.https://aur.archlinux.org/packages/nitter-git
In case anyone uses Arch, the AUR package includes the @cmj patch to disable the expiration check, so you can use you tokens, see #1164.
I'm also running this with my very old personal account and it works well for a personal instance with a few users. This seems like the best way forward.
you don't have any problems with pagination? if you go to page 4-5 of a tweets search?
Works for me, with some rate limits, using a "real" account.
How exactly could you add your own account to nitter again?
I think there are different ways to go about doing it but unfortunately all of their instructions are buried in this and other threads. Here's what I could find:
#1164 and https://github.com/zedeus/nitter/issues/983#issuecomment-1914616663 worked for me.
The videos no longer work, though.
Could someone write a tutorial for setting up nitter for a local rss reader using an account?
@TempUser13 Glad you asked :) I was working on it exactly and had it here https://github.com/sekai-soft/freebird
I was able to run a private Nitter and miniflux instance locally on my NAS and had been using it for a while without trouble.
this doesn't help anybody at all, it's just a way for you to self flagellate
@TempUser13 there are guides here but utilizing your own account is super easy using a python script in here (sorry to the author i dont care to look for the source now it's literally buried now)
basically:
Message over at the nitter-mirror https://n.opnxng.com/ :
Links from above message:
https://arstechnica.com/tech-policy/2024/02/twitter-front-end-nitter-dies-as-musk-wins-war-against-third-party-services/
https://liberapay.com/zedeus/
https://soundcloud.com/trueanonpod/sets/the-lamest-show-on-earth
not over. I've been running nitter both publically and privately with accounts older than 2016 with great results. nitter run as a free service might be over but some operators who choose to run it at no cost (or by donation) it's 100% viable.
I also ran a week of tests with 25 suspended accounts -- you can use suspended accounts with nitter but graphql randomly blocks (or maybe nitter doensnt handle rendering them). in my tests using 25 accounts I could see:
disclosetv: 4
bnonews: 11
bnofeed: 17
elonmusk: 12
nasa: 21
suspended accounts can see all of these users, suspended accounts can be used to view timelines normally on nitter but graphql is limiting viewing some accounts and i haven't figured out why yet.
@zedeus I have about 100 guest accounts from 2022-2023 that are just suspended accounts, you wanna meet up on matrix and check out what im talking about?
example of public facing with aged accounts - avg requests are 4-8000. limited accounts have only been maybe 10-20 max daily. they reset every 12 hours
multi works: https://nitter.poast.org/nasa,cnn,msnbc,twitter/
pagination past 4 pages works. you just need >15 accounts to make it work. this seems like a nitter limitation more than twitter maybe?
final edit: i forgot to mention I tried this with accounts from 2022-2024 and accounts from 2010-2016 and the limits for the latter are almost nonexistent. 2022-2024 accounts have maybe 10x the limit of the guest accounts -- 15 of them would burn out publicly in an hour on the same server im running 72 2010-2016 accounts on daily burning out 20-22 accounts total
I assume that mirror is private since Im getting this when trying to connect:
not private. just heavily restricted. you're either visiting from an IP I've blocked or you are using an out of date browser
How would a website know if my browser is or isn't "out of date" whatever that means?
The User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.
So it's a wildly inaccurate guess
@animegrafmays On the "guest_account" branch, do the videos work? What's the specific commit you are using?
That's the only missing feature I can tell with a "real" account.
using the guest accounts branch here, the latest commit. nothing modified. videos and multi only work with more than 5 accounts. as far as I can tell it's related maybe to how the data is fetched
here is an example: https://nitter.poast.org/NASAUniverse/status/1758594049991225404
our nitter currently has 74 account tokens from accounts aged older than 2016 as newer accounts have much worse limits
edit: just checked with a private instance I run for our fediverse bots that has two accounts and videos play so the limitation must just be 2
That video works fine for me using 1 real account just as long as proxy video is disabled.
@animegrafmays it's a cool discovery, but I wonder how feasible it is. I'm not sure crowdsourcing suspended/old accounts would particularly successful, and Twitter would likely close the suspended access loophole.
/.healthshould give you some rate limit info, but your instance returns 403.the people who use my fediverse instance originated because they were suspended on twitter, im sure out of 31k users we could drum up quite a bit but for the normal person i agree it's probably not as easy as say spinning up a new account
this was oversight, you can view it now. anything i can do to aid in facilitating the use of nitter by other operators let me know, I'm open to whatever
edit: the instance domain is actually pointing at two servers so to get a proper estimation you'd need both
https://nitter-west.poast.org/.health
https://nitter-central.poast.org/.health
I'm surprised your traffic is so relatively low with most instances having died, nitter.net was getting 8-12x more 24/7 for months leading up to the guest account creation removal, despite heavy anti-bot measures.
combined it's about 12-14k avg. I've been heavy handed with blocks. I had preemptively blocked the status page and blocked github referrals (removed it earlier today for the example post above) so it's mitigated some of that hand off traffic. have lots of ranges blocked from previous scraping as well
I blocked every cloud range and actively blocked abusive ranges and IPs, plus many user agents and all requests without standard headers, but it was never quite enough.
I also used fail2ban to ban IPs hitting nginx rate limits, and a custom setup to ban IPs not fetching static files served by nitter. Helped a lot, but traffic climbed back up.
these are at minimum /24 but most are /16. that config also doesn't include amazon, oracle et al:
I've been working on this for a little over a week
Blocking Tor wasn't a goal, so that's fine. I suspect a lot of traffic came from mobile proxy providers.
@stopmotio my personal experience as nitter operator matches Zed's: Tor is a small portion of traffic overall and entirely insignificant when it comes to abuse. I had the data to figure that out. Even if 100% of my Tor traffic was abuse, I would not have cared because the volume was so low.
The load from cloud providers was by far the most significant (particularly AWS doesn't care what people do on their servers) and yet also easiest to block. Residential proxies/mobile proxies/botnets were a close second, and much harder to block.
I ran the same mixture of heuristics as Zed, and the biggest problem were mobile proxies and things that looked like it. Your regular abuse/DoS rules don't apply here, and so for example Cloudflare does not help much without additional config. Scrapers are specifically written for Nitter instances, and unless we're talking about showing CAPTCHAs for every user, the abuse prevention had to be as well.
is it dead? I get error but didn't really refresh it that many times (I assume most people are trying to use that instance)
Edit: I refreshed it and it worked. not so sure about the video but that status is displaying with its replies too
I'm honestly not sure why one would block of a US ISP's IPv4 addresses (at least in my case). I'm pretty sure I'm using a recent version of a chromium based browser....😕
there's no AAAA record so it has nothing to do with IPv6 as it is IPv4 only.
I'm honestly not sure why one would block an entire ISP's IPv6 blockone of the largest DDoS botnets runs on Hetzner and almost exclusively abuses their v6 ranges. thats a perfectly valid reason for doing such. but this is not something that has been done on this nitter instance.
(Apologies all for the slight segue here.)
My bad. I assumed your Nitter instance was dual stack (I've edited my previous comment to correct for that). I'm still not sure why you would restrict/block access to your instance from a block of IPv4 addresses from a major US ISP. I personally think its a bit brute force for ISPs, but that's just me. Then again it seems like your services attract quite a bit of hostile traffic (https://github.com/zedeus/nitter/issues/983#issuecomment-1953328407)....
I'm sure that is possible.... though the end-user experience would probably be quite poor for a public instance with the traffic volume of say @animegrafmays' instance. That and you're just playing a game of cat-and-mouse with Twitter/Xitter. It's not a long term sustainable strategy. Private instances seem to be the future. One person mentioned a guide to deploy an instance to a Synology NAS: https://github.com/zedeus/nitter/issues/983#issuecomment-1949859380, this PR #830, and this Nitter forks pre_guest_accounts (https://github.com/PrivacyDevel/nitter/tree/pre_guest_accounts) branch.
A private instance may not be useful if the point is to avoid having a Twitter account. I will simply stop trying to read Twitter timelines at all if I cannot get them in chronologicial order without an account. Elon Musk can clean my toilet.
Twstalker still works, amazingly. Is it because very few people use it, and it gets under Musk's radar?
Has anyone tried and had any success in automating user account creation? Seems like that could help public instances survive but the captcha seems like a big issue...
Not the best place to say it but thank you for keeping nitter.poast.org running, seems to be the last Nitter server now and we desperately need it (I run @splatoon@wetdry.world on fedi).
Still works really well, I'm not sure how we would be continuing on without your service (I assume bird.makeup isn't doing that well lol).
he was still scraping our instance up until a couple days ago. I asked him publicly why he was still doing it and it seems to have stopped
if you reach out to me on there I will give you access to a private ip-whitelisted instance I run just for bots, that way you're not limited by the current measures I've put in place on the public one
I am still using nitter.net, with Firefox set never to override user choices in overriding the certificate error. Since I am not logging into anything and those are public posts I am reading, an MITM attack isn't worth much.
Can anyone with a private account actually watch videos on nitter? If so, what's the exact commit you are running to perform that black magic?
Videos should work again with the latest guest_accounts commit.
I tested earlier on nitter.poast.org with a video on NintendoUK which had the latest Splatoon song lol
Not sure exactly how you're doing that when nitter.ner has an expired cert + HTTP Strict Transfer Security (HSTS) enabled.....😕
he explained how in his post. im starting to question your username
You do understand that your GitHub issue has roughly 683 comments, right? I probably did read @lukefromdc's comment at some point. Him or someone else is using another tool to scrape those instances.
Let's not go down that road....and stay somewhat civil?
sure, you are replying to a public issue i created and every time you do that i get an email. in an effort to try to aid the issue i raised and to prevent getting multiple emails about the same issue or comment i try to be as descriptive as possible. getting an email at 3am from you that says "Let's not go down that road....and stay somewhat civil?"
I would not even know HOW to set up a scraper. Rather I use nitter instances to manually monitor remaining Twitter timelines from my community. This is because of Twitter's refusal to show chronological timelines (the only kind useful to me) without an account.
Again: to get past the certificate issue in Firefox (do NOT login to anything showing a certificate error!):
1: open about:config
2: set
network.stricttransportsecurity.preloadlisttofalsein my experience over the last week it appears twitter limits based on account+ip. using the same list of guest accounts on two dedicated server they are always different. right now there's 11 accounts limited on one server and 6 on the other. i wonder if we can use this to our advantage, perhaps a shared pool of accounts would actually work
There are tons of websites that list public proxies in residential areas. 90% don't work, but if you can sift through them they're very useful for scraping.
sure. being listed on that page is a death sentence. the warning on the top of the page begging people to not scrape instances is ignored. being scraped is bad. so the status page is blocked so we don't show up. i still also aggressively monitor logs for scraping attempts and have blocked entire AS ranges from major providers (google, aws, digitalocean)
Twitter recently broke videos on Nitter. Can somebody please fix them?
try this: https://github.com/zedeus/nitter/pull/1178
Can confirm
cdff5e9works, thanks a lot.@zedeus #1164 is still needed with older accounts.
@somini I coincidentally merged that right before you commented
Oddly, I can only access your instance through a VPN, despite never having used it before. Do you block certain country IPs or dynamic IPs? Or is something wrong on my end?
(not OP)
if three or more IPs in a /24 range have been blocked by fail2ban the entire /24 will get blocked. you can email the first two octets of your ip to graf at poast.org and i will look but that's more than likely what it is
Uh... That's far too advanced for me, I'm afraid... I don't even know what a "24 range" is... Should I email you my current IP?
that is what i was asking for, yes
Ah, "Graf" is your username! Still, I'm too dumb and couldn't find a way to contact you directly, so I just sent an email to Poast support.
If we have reached the point where instances have to carefully ration requests:
@zedeus how hard would it be to implement caching/database storage of Tweets?
Most likely a significant portion of requests are for the same (popular) Tweets over and over again. If instances store or cache Tweets, they don't have to spend their request budget getting the same Tweets repeatedly.
Yes I did check the other issues, the closest I could find is #288.
Even more useful would be a collaborative or shared cache used by multiple Nitter instances. This would effectively pool their accounts and request budgets, without disadvantaging any instance operator or requiring them to sacrifice request budget to help others. (EDIT: nevermind, #1188 discusses this)
I have been told some instances are caching certain users but I've not been told how yet, so I would be for something like this. maybe 2-3 requests in a short time would suffice to trigger some kind of caching. you could probably do it with nginx but im not sure how you would cache the data returned that way. I've been caching twitter media on a 3h expiry and it's helped a lot
@animegrafmays Yeah, it seems like caching was once done more extensively but reduced for performance reasons (unfortunately I can't find the issue anymore). Now since it's a choice between slow loading and not loading, even a very large and not very performant cache is better than being unable to load Tweets.
This might also help a lot in dealing with scrapers -- they are probably some of the worst offenders in requesting the same major accounts.
Twstalker is doing funny things: when someone quotes a tweet, the quoted tweet repeats what the quoter says.
@animegrafmays Hey, how/where can I contact you to whitelist my IP for nitter.poast.org :) ?
im not in the market of 'whitelisting' IPs. you can email me at graf @ poast.org and I can see why yours is blocked, but I'm not whitelisting people
@animegrafmays Mine is blocked too, maybe because I use the RSS feeds for some accounts (~ 10). I set them to fetch new posts once per hour on my browsers, instead of every 30 minutes. Before I e-mail you, could you confirm that it could have been a cause of “ERR_CONNECTION_REFUSED”?
this github issue is not an issue tracker for your connection issues to my nitter instance. please stop shitting up this github issue by triggering an email every single time you respond and instead use the contact methods i've presented in this thread already
I have never had an issue with the RSS feeds lately, I get them fine with Power Automate and Element Feeds.
Fully agree with the fact we are clogging up but it's not like I can move this to fediverse (the servers I am on block poa.st because of allowing some undesirable people that were on certain platforms that's all).
The solution to all your problems is to host these services yourself. One of the common SBC or an old laptop - screen not required - is more than enough to run an instance of Pleroma (for your fediverse needs), Nitter, Libreddit, Invidious (Youtube proxy) and even Peertube if you go easy on the transcodes. That way you are neither a burden to others nor beholden to them. You may get blocked by the 'purple-hair crowd' on Mastodon but that is more or less unavoidable unless you want to become part of that crowd yourself. As long as you behave sensibly - and why wouldn't you? - you should be able to federate with the world and its dog, minus the aforementioned block-happy group.
Of course all this assumes you either have a stable internet connection at home to which you can attach said SBC or laptop or thin client or what have you or can splurge a few
${currency_unit}for a lightweight VPS somewhere. It does not take much in the way of technical prowess to get these things to run, you can even go the way of e.g. Yunohost or CapRover although you'll probably soon get tired of their restrictions and lack of flexibility.For those of us with no landline, a need to self-host a service simply to access Twitter content is the same as Twitter not being usable.
Wow, amazing information. If I was able to, I would have done so already. I have no experience with hosting stuff and there's no chance of me running up a cable to connect a device.
Using a VPS seems expensive too, a local company wants £3 for their lowest tier a month.
@lukefromdc how is a landline related to this?
If you only would use the machine for TwiXXer->Nitter, yes that would be overkill. The idea would be to use your own host for much more than that both to avoid the mentioned blocking problems as well as to host your own services. Host your own photos instead of giving them to Google, host your own music instead of being dependent on streaming services, host your own XMPP server instead of being dependent on Telegram/Whatsapp/Signal/Matrix or (
${deity}forbid) Metafacebook or Apple or Microsoft et al.Landline relevence is much more bandwidth available. I have de facto (not admitted to) capped cellular bandwidth via tethering only, when I am on the road it would be out of range of any home server. Forget a static IP address of course.
Also they don't like tethering but blocking sophisticated ways to tether brings lots of false positiives so they don't bother. If they do I get a new carrier and a new phone number. No stable IP address, no continuously on connection in any single location, not going to carry a server everywhere I go and we are not discussing the presumably lighter load of distributed hosting (e.g diaspora) here.
I do not however use Google or Meta for any purpose as I oppose their business models. I publish videos on Mastodon and on archive.org (latter good for linking especially as they want copies of all online content anyway). Twitter used to be widely used in my community so I had to be able to keyword search it and pull up known useful timelines, now neither of those work on the Twitter webapp without an account. I refuse to make an account with any ad supported corporate social media site, but the importance of Twitter is plunging as people stop posting there.
Unfortunately this is fueling a migration to Instagram which is even worse than Twitter but at least still has a partially working 3ed party backend. You must know at least the username whose content you want to see to use it as far as I can tell. On the plus side, the migration to Mastodon has greatly boosted the reach of my content, but some events I would cover now go unseen as they are organized in places I am now blind to.
Ah that makes sense.
Providers here generally don't require a landline but one provider still only does ADSL service and the other is HFC.
so people stop generating multiple notifications via here and email for stuff that isn't related to nitter development, if it pertains to my nitter instance please email graf at poast dot org. do not post it here. want to be critical of me? email it. have a problem with what im doing? email it. leave this thread for dev.
thanks
Cheers for all the work that you do on Poast bud @animegrafmays
Apologies in advance for the plethora of questions but I would appreciate some further tips before I start smashing my head against a wall trying to get a private instance working well enough for my needs. Given your experience running the most reliable Nitter instance as of late I would be very grateful for any advice around accounts usage and I'm sure I'm not the only one.
Basically I need to scrape a few hundred accounts, their complete timelines including likes, complete list of followers and following. What would be the best way to go about it? Would it be more effective to whip up some quick scripts that query graphql rather than run a private nitter instance?
I'm not as concerned about performance gains as I am with how nitter cycles through the accounts. I wonder if e.g. requesting the timelines in order with sane delays through some custom scripts would help as far as not getting the accounts blacklisted as much.
I've also thought about queuing up all the tweets containing media and then spinning up a headless browser separately in order to request them directly from twitter, since you can seemingly still do this much unauthenticated. Or is this overkill?
You've mentioned aged accounts faring better but most people, myself included, don't have access to any. Although the scale of private instances is infinitely smaller than the one you run, I'm still wondering what measures I could take to optimize fresh accounts usage.
Would running each account through a separate fixed residential proxy make a big difference or is it not worth the hassle?
Similarly, would using really big delays make a difference as far as not getting the accounts blocked in any way? For my particular use case, performance is not as much of a concern as getting full coverage. I remember reading in another thread that someone running a private instance for testing was noticing several accounts not retrieving full timelines in some cases, with random tweets missing. That would be a big problem and something I'm looking to avoid at all costs.
Thank you in advance for any insight which will result in less hair pulling on my part.
i run private instances for this so they don't have impact on the public instance, you can reach out to me at graf [at] poast.org if you are interested.
for what you are asking, if you are doing a one-time thing, running a quick nitter with some even new-ish accounts would be fine. the limits for accounts older than I think 2018 are doubled whereask those prior to 2012 seem to be tripled or maybe quadrupled. i can't find documentation on this but in practice we run 200 accounts older than 2012 and very rarely run into limited accounts beyond maybe a dozen out of that in a day. older account limits are much, much more lax and definitely the way to go if you can get your hands on them
if you are using RSS it's limited to the cache time of the nitter instance. nitter.poast.org is on 30 minute refresh for rss, staggering requests on a private instance (i.e. request 5 accounts now, 5 more in 5 minutes, etc) would help spread out user account exhaustion but in my experience as I said prior you can more or less ignore this with aged accounts. banned accounts have limits almost identical to those created in the last 12 months so if you can get your hands on a bunch of tokens from those and you're just serving yourself/your own interests its more than adequate. it's a lot easier to find banned/suspended accounts from people than current, active ones
you mean fetching a list of attachments on the tweet and bulk downloading? seems a bit overkill. if you are running a private nitter instance you should be fine without needing to go to this extreme
we happen to have a relatively large userbase (32k users at time of writing) who had donated a bunch of accounts and I have some that I had left over. I also have some surplus I could give you tokens for if you're using a private instance for yourself to get you started. we have 22-25k requests/s avg (just recently had to move it to a ryzen 7 server because the core clock on individual threads wasn't enough even running multiple nitter processes)
to be honest, as long as you aren't listing yourself on the wiki or on the status page you likely wont run into issues. new accounts have about ~200 profile queries each that seem to reset between 8-12 hours, so a private instance with maybe 5 or 6 accounts would do you with staggered scraping for quite some time
i dont want to clog up this issue or notify everybody so if you'd like to discuss it further you can email me at the address above
Why are you even engaging in a convo with someone that "needs to scrape a few hundred accounts" and then offering to help with his endeavour? I must be taking crazy pills without knowing.
Twitter's fault for not providing an API for this. There's nothing wrong with scraping.
I'm not going to take your bait and get bogged down in a pointless debate re the ethics of scraping and how it all might pertain to Twitter's newest policies.
From a pragmatic standpoint, several Nitter instance maintainers have mentioned in this very thread that scraping is a big enough problem as far as keeping Nitter usable for regular users. Since I prefer to use Nitter myself for casual lurking rather than having to deal with Twitter's interface and algo, the last thing I wanted to do was contribute to the problem.
I believed addressing this out in the open would go a long way towards helping those with this specific need get their own private instances up and running, rather than ruin a good thing for everyone. Taking my own use case as an example, since speed is not a concern it would've certainly been more convenient to scrape graf's instance for weeks rather than have to deal with Twitter socks, but it's a really shitty thing to do.
As for why someone might be interested in scraping Twitter on a small scale - several hundred accounts is nothing really despite your scaremongering - there are legitimate reasons. From Twitter's shitty built-in search to archival purposes; some communities sharing precious tidbits on Twitter are already more or less shutting down in protest because Elon man bad.
i'll break it down for you -- i'd rather somebody scrape an instance i run that nobody else is trying to use than use my main instance, that literally everybody is linking to. cool with you? no? i dont give a shit
Scraping is why we cant have nice things such as Nitter in the public as it used to work before "some" started to scrap Twitter through Nitter and put Nitter to a halt.
nitter.poast.org sometimes doesn't show search results or individual Tweets. nitter.privacydev.net shows the rate limit error on all searches and account pages even with a full bar of green at the Nitter status. It'd be great if people started working on a solution as opposed to dooming every moment about the death of Nitter 😐
the reason searches are "sometimes" not shown is due to the amount of accounts limited. there's currently 200 accounts on that instance and because it's serving traffic to almost 200 thousand unique visitors daily more than half of them are rate limited. with that amount of traffic having more than half the accounts rate limited will result in not being able to expand tweets, searches not functioning etc. it's a literal full time job maintaining it and preventing scraping but I am doing what I can including procuring an additional 100 to add to it this week. sorry I can't be good enough for you, @nkfm200 😭😭😭
@animegrafmays That I've heard of, the small number of accounts serving a huge number of visitors every day causing rate limits on accounts. Hopefully that extra 100 accounts is there at the time when it's the end of the week. This may be off-topic but I'm just strict about stuff because this is a world where the people living in it refuse to do something unless it fits the rules of the culture
hello, I have opened a new instance with around 500 accounts. feel free to try: https://xcancel.com
Most popular RSS readers should also work on that instance. If you can't use it on your favorite rss reader then please send me an email rss [AT] xcancel [DOT] com
Thanks.
@unixfox saw this. mad work, thanks for sharing.
fellas, looks like twitter backpedaled something. instances on commits from back last August (before the guest account branch) are working fine
https://t.com.sb/dominickmatthew/status/1802031639792632297#m
can confirm, I just rolled fresh deployment of
github.com/zedeus/nitter@d7ca353with no guest accounts and it is working. I will have my users use this instance and see wat kind of limitations they've imposed and report backhttps://nitter.nerv.st
search doesn't work, media tab doesn't load but going to specific users timelines, viewing their threads/comments are all fine.
edit: seems like they've reverted it again
{"errors":[{"message":"Bad Authentication data","code":215}]}funny timing, I was also looking into why my code from a year ago was suddenly working lmfao
i can't claim to know what they were or are doing, but i was able to spawn a pre-guest account branch version of nitter and have it work reliably. here's hoping maybe they fully reverse their choice but at this time nitter is still "dead"
Looks like they switched back on the old v1.1 timeline endpoint? It's working on a bunch of old instances right now...
it looks like they fucked up something again. my old instance, running 2023.08.08-d7ca353, suddenly started working again.
this is definitely what's happened, they've turned it on and off randomly over the past several days so i was hesitant to comment. i don't want to speculate but I also don't believe they'd accidentally enable it again. bait, maybe?
Now 'all' that needs to be done is to get video playback working again. TwiXXer videos can be played using 'any' media player, e.g.
...shows you some knuckleheaded adherents of the climate doom cult putting their tribal mark on the Stonehenge monument. The solution should be fairly simple, I might even have a look at it myself once times arrives.
There is no need to mocking people's irrational action to a grim future.
Looks like v1.1 API opening up was temporary
https://status.d420.de/
Most instances are back to being down.
All this great stuff I'm missing because I'm not on Twitter...
<rant class='off-topic' intensity='sizzling'>The future is quite bright as long as groups like JSO/XR (UK), Letzte Generation (Germany), Återställ Våtmarken (Sweden) and similar cookie-cutter activist collectivists are held in check. These are doom cults which, in contrast to the doom preachers of yore with their sandwich signs 'The End is Nigh, Repent!' who only rang their bells and shouted a bit use destructive tactics to proclaim their quasi-religious zeal. They do not help whatever cause they claim to adhere to, if anything they achieve the opposite. If you believe in the climate scare - which I do not - and want society to move away from fossil fuels - which I do, just not for those reasons - you should not root for these groups.
As to that 'grim future', do you have children? Do you think it a good idea to tell them they won't live to reach the age where they might have children themselves? Read up on how children react to that type of rhetoric, especially younger children. They actually believe it word for word, abject nonsense though it is. What a way to portray, nay destroy their future.
So yes, call those numbskulls for what they are.
</rant>I take this as you also don't believe in modern medicine(which use even worse methodology than those climate "scare") and never go to hospital.
Which is fine by me. Keep going. You are doing good for your children.
Keep this on topic, please.
@ImTheDeveloper I've been watching this over the past 10 days. It seems all the endpoints were flipped on by accident last weekend after they realized the still left the v1.1 likes rest endpoint running since ... always.
For about a week (until Friday the 21st) they had all timelines open and various other endpoints that enabled old Nitter to operate.
My observations from this: https://gist.github.com/cmj/bdadb2c324e2022fffb2cf2ca0c3e7b8
See: https://github.com/BANKA2017/twitter-monitor-assets/tree/master/rate_limit
Plenty wrong with it. Most won't admit that they use scarped data, that's one thing wrong.
Asking is an act of respect.
I don't respect twitter
Thank you for your personal opinion. I am neutral to Twitter and don't have an attachment to what it was or may have been.
Now when you say , I don't respect twitter, aren't you just in the deficit? Shouldn't they say I don't respect nukeop or their work.
You're practicing the same attitude these tech companies you love to hate.
perhaps I am wrong and am supposed to act out in some way, but I'm not attached to it, I don't want to work for them, avoiding tech work that has anything to do with predatory practices (yes I do and will lose on job opportunities)
I just want to scrape whatever I want and they should be obliged by law to allow me and others to do it.
You now fully sound like a child, this isn't judgement against children at all and not a judgement against adults that chose not to process their issues within. You are or might be a talented engineer, but you bring your unprocessed with you.
If we decide that X/twi/whatever is running an unethical platform with unethical policies, we don't need to act out. maturity damn it
@lukefromdc Hey Luke,
When was the last time you've donated or supported x.org?
Go ahead and organize a bunch of people, take time , take a year and figure out seeing Musk.
You have all the choices to bring up an issue.
We were talking about X , not elon musk (don't care about specific individual) and their product.
I have a feeling you have a difficulty staying on topic (so very common now, I'm guilty too at times). Forget Oracle and are you a common person directly affected by Oracle? I met James Gosling back in 2005 sun conference. I'm sure he wasn't happy about Oracle but continued his craft. Continue your craft or volunteer.
all that being said, I have no negative feelings toward anyone
this conversation is only tangentially related to nitter, can you stop shitting up our email inboxes?
Sorry about this @animegrafmays. Stopped
Now that the moronic-talk is over. Questions about Nitter, specifically with regards to the regular accounts version.
For those that don't know, the guest_accounts version used 2 curl commands to generate temporary accounts, however there was recently a fork (which I can't find right now) that used real accounts instead.
Any info helpful, thank you!
@RunlevelConsulting literally the 7th result after typing "nitter" in the github search: https://github.com/sekai-soft/guide-nitter-self-hosting
out of nowhere my instance started working again, anyone else?
Mine never stopped working (except for video, that has been broken for a long time) but I'm using it with a real account, would you happen to have it configured with some of those as well?
Yes, I'm working with some real accounts, but I remember that stopped working too, I used my own instance for scraping, now I migrated everything to a new system, I'm glad that nitter is still working.
Replying to myself for those who also have problems playing video: it does work when you disable Enable HLS video streaming and proxy video streaming through the server while keeping Enable mp3 video playback enabled in Settings (the cog wheel in the top bar, top right). In this case you get mp4 video files which can be streamed directly in the interface.
If you're using Firefox you may need to disable 'Enhanced Tracking Protection' since it (rightfully) considers
twimg.com(where video content is served from) a tracking domain and will refuse access when that setting is enabled. With these changes video is fully functional in Nitter.It is also possible in some cases when twimg/twitter JS is disabled to click on
a link to the video itself that will show, then download the file. For some folks
enhanced tracking protection is mandatory and cannot be turned off. Fixing that
would require a way to create an exception in firefox itself to allow twimg.com
from a nitter instance.
You can add per-site exceptions for ETP (Enhanced Tracking Protection) in Firefox. These exceptions are all-or-nothing so it is not possible to only let
twimg.comthrough for the whitelisted site. This could be made to work if Firefox allow users to state an ETP level - off, standard, strict or custom - for whitelisted sites but that is currently not the case.That might be enough so long as nobody ever deploys an ad-supported Nitter instance
Whatever it might be worth here is a working set of settings when using https://xcancel.com/
Display
Theme: Twitter
Infinite scrolling (experimental, requires JavaScript): Enabled
Make profile sidebar stick to top: Disabled
Support bidirectional text (makes clicking on tweets harder): Disabled
Hide tweet stats (replies, retweets, likes): Disabled
Hide profile banner: Disabled
Hide pinned tweets: Enabled
Hide tweet replies: Disabled
Square profile pictures: Disabled
Media
Enable mp4 video playback (only for gifs): Enabled
Enable HLS video streaming (requires JavaScript): Enabled
Proxy video streaming through the server (might be slow): Disabled
Mute videos by default: Enabled
Autoplay gifs: Disabled
Link replacements (blank to disable)
Twitter -> Nitter: xcancel.com
YouTube -> Piped/Invidious: youtube.com
Reddit -> Teddit/Libreddit: teddit.net
Only drawback so far is that the replacement of youtube doesnt work in a post, only for quoted posts.
And that loading too many pages at once will throw you into some kind of temporary blacklist at xcancel.com (so have at least a few seconds between each load/refresh if you visit more than one account).
Also Im not sure on how multiloading usernames affects getting onto the tempory blacklist.
That is when you load multiple accounts in one go such as https://xcancel.com/elonmusk,cb_doge,joerogan,nsagov (just picked some random accounts as example :-)
tonight during the voting in the US presidential election we topped 1.21 million unique visitors on nitter.poast.org. right now, during the counting we are averaging 21 thousand requests per second across 4 ryzen 7 3800x load balanced.
this is insane. and the 400 accounts we are using isn't even close to enough. i offloaded a lot of requests via proxy directly to twitter via a separate proxy_pass. helped massively. never been more proud to put so much effort into running it 🙏
thank you for everything @zedeus, i think everybody here is thankful for your efforts
I have been using https://nitter.privacydev.net/MarkDavis/ to keep an eye on the one twitter user I follow as I much prefer the Nitter presentation.
For the past three days I have been getting nothing but "Instance has been rate limited. Use another instance or try again later." messages when I attempt to check.
Did they break something new, or am I the only one seeing this?
works fine for us -- https://nitter.poast.org/MarkDavis/
might be that the instance you are using is genuinely rate limited. not sure how many accounts he is using, but on our instance we run 4 dedicated servers in round robin with 100s of accounts, so usually reloading on ours fixes that issue if it ever appears for you.
this isn't Xitter changing anything or nitter being broken, it's related to rate limits of accounts used to display the tweets to you
Thank you for your feedback. It does indeed appear to be a problem with that instance in particular. I have changed over to xcancel.com and it is working well. I had though the instance I was using the only functional one and was pleased to discover I was wrong.
you're welcome friend. im glad you found a solution that fits your needs
@dhdurgee here is all the instances that works: https://status.d420.de
Im experiencing the same issue as you since few days now. However im using Tor and none of the other current working instances including xcancel.com allow Tor connections which is very sad.
https://nitter.privacydev.net/ was the only instance that you could use to browse with Tor.
I hope the owner can get it back up or hope other instances can allow Tor at least
I made the decision after several days of heavy abuse from TOR exits to block them. I will not be removing the block and have automated updating it with any new exits. prior to blocking TOR exits, we were receiving on average over 30k requests per second. after it's down to 10-11k. all used for scraping. it's unfortunate that people abuse TOR in this way, but it's not something I will consider removing.
@initd0
Odd xcancel.com works for me on Tor with the default settings.
Can you send an email at antibot [at] xcancel.com? With some explanation and a screenshot.
I'll try to see what's going on.
My guess is the few surviving Nitter instances have to take extreme precautions against
bulk scraping expecially by bots or AI to avoid being blocked.
If someone wanted to set up a dedicated (maybe .onion?) nitter instance just for Tor,
they'd need a way to rate limit each connection to human possible numbers of
connections. This would have to be for each connection separately, and at a very
low rate like one request per second. DDOS scripts might be possible to break by
having the whole instance respond to a wave of bot hits by going offline for first
30 seconds, then five minutes, then an hour and escalating as needed if attacks
continue.
I'm sure this has been answered but as it's now an enormous thread, it's hard to find the details:
What is the canonical way to configure a nitter instance using multiple accounts for use, ideally with tokens being refreshed if needed? Thank you.
https://github.com/sekai-soft/guide-nitter-self-hosting
If you can just whitelist QuiteRSS reader, that would be great, because it throws an error saying "RSS reader not yet whitelist!" - I think thats the only issue preventing from using Tor in the reader.
The actual onion links on the wiki dont really work either.
Please follow the instructions and send an email at rss [at] xcancel.com
This should now work for people wanting to add accounts by hand.
https://gist.github.com/cmj/998f59680e3549e7f181057074eccaa3
Looks like xcancel.com have been malfunctioning for a few hours... hopefully that will be resolved soon?
"
500 Internal Server Error
openresty
"
The main page at https://xcancel.com works but not any subpage to get actual user content.
As of now nitter.poast.org still works
What can I do in case of the account has 2 Step Verification?
I'm happy to announce nitter.net has finally returned. Instability is expected for a while until scraping prevention is improved, etc.
Running an instance is now both difficult and expensive, but it's possible. If anybody is interested please feel free to message me on Matrix.
I'll be updating the guest_accounts PR soon and merge it to master, stay tuned.
Good to hear, thanks for sticking with us friend. Looking forward to many more popping up soon 🙏🏻🩷
Good to hear, I've been running an instance all through the changeover from Twitter to X, from no-account to guest-account to registered-throwaway-account and will be glad to go back to guest-account.
Sorry to disappoint, but guest accounts are not back.
I'm not a developer. I just want to express gratitude to this project and may I know why they win?
What are the minimal requirements for get this working? It's about time, money, technical knowledge or all of them at scale?
Running a single instance is the best humanity can do those days? I'm polite asking, trying to understand and giving praise to devs.
It's currently the same on Invidious and similar teams. Free internet is almost dead?
Please delete this message if it's not a valuable comment. Trying to philo it's the best activity I can share here.
better late than never.
@csisoap , these first few lines seem to work just as well.
requestsknows how to do basic auth/base64 encoding...i'd probably rename
bearer_token_reqtotoken, and its values are strings, sosince the default content-type is urlencoded, if u set data correctly:
also, you can probably
and then remove
headers=twitter_headerfrom future calls tosession.post()Closing, any further discussions may take place in #1212