18+/"sensitive" tweets no longer available #829
Labels
No labels
accessibility
bug
discussion
documentation
enhancement
feature parity
good first issue
help wanted
question
rss
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
PAS/nitter#829
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Any 18+ Tweet that could be viewed before can't be any longer, did some update age restrict them? If so, then how do I switch it off?
I believe this may not be due to a Nitter change. There's been a change Twitter has made that places 18+ content behind an age verification check. Since there's no way they can verify that users (that consume content from their API) are of appropriate age, they just dont include them.
I can't believe that that's entirely correct, I still see some thumbnailing services, such as vxtwitter, fxtwitter, still functioning in this case. That said, I also note it started occurring on nitter.net itself a few days ago while it only seemingly just started hitting some mirror sites, so I suppose it's possible that it's a slow rollout, but it's still also possible these other sites are doing something that nitter isn't
IIRC based on some warnings I've seen when using yt-dlp plus other stuff:
Edit: nitter.net uses more graphql endpoints than most instances because it's running a dev version
Will the workaround be added to the browser instances?
Idk,.
It seems like Twitter's API has changed
I haven't found one instance that works now.
I tried many of those not updated.
That is definitely the case. I tried them all and a few unofficial ones. And some of the ones I tested have not been updated in months.
Mine has my workaround already applied and works tho.
https://nitter.privacydev.net/
How does the workaround… work? Whose credentials are used?
The administrator of the respective instance needs to create a Twitter account, sign in and then put their session information into Nitter's configuration file. This session information will then basically be shared for all users. It's the instance's Twitter account, if you will.
As an end user, you shouldn't need to do anything.
Won't that lead to severe rate limiting, though?
It seems like that does indeed happen sometimes. But for the most part it seems like it's actually working fairly well for my instance. But I am not yet aware of any other satisfying workaround for that problem. Forcing a user to login and get tracked is not really a satisfying solution for me.
I see the following general solutions for this problem, personally:
Luckily, the workaround works but it's unfortunately very slow.
indeed, slow. probably rate limited already I guess unfortunately.
I am waiting so far 5 minutes for a vid clip to play. gonna have to give up
At least the tweet comes up tho,
A+ on effort
TYVM
What about a voluntary system - ask for a login to increase the rate limit, but fall back on a set account if no login is provided?
I use Nitter mostly for its RSS feeds. Since I have a lot of NSFW accounts as feeds, some of whom tweet frequently, the frequency by which I ask for tweets is probably a major reason why the instance is so slow. If I could somehow pass my own credentials when fetching those RSS feeds, even if it meant I was tracked, I'd do it if that meant a faster or more reliable system with less caching for all.
Perhaps users willing to forgo the non-tracking nature could put a query string at the end of the URL for the RSS feed, containing the exact ct0, auth_token and csrf-token, and the server could use those credentials instead of its own if all are present.
I just made it so that videos aren't proxied anymore by default on my server to take some load off of it, make it so that people can actually watch videos again on my instance and to hopefully make Twitter not rate limit my instance as harshly.
And I also made it so that the request of userIds is made unauthenticated. Hopefully that makes it so the instance doesn't get rate limited as fast anymore as well.
A URL-based solution would work the best for my needs, if possible.
Thanks for that
Now working flawlessly
Hopefully it can last but appreciate what you're doing nonetheless!
The search tab still shows the complete feed. Could fetching this way be a workaround? However, it seems to depend on the region again, e.g. US works, DE does not.
That bizarrely works (not with nitter.net but with other instances at least) and adding the "/rss" at the end cleanly enables rss functionality lol thank you very much for this.
I understand that the solution that is being proposed will probably only be applied by a minority of instances, right?
I see a problem and that is that the +18 tweets seem to disappear completely. I don't know about the web, but in the RSS feed there is not a single reference. This implies that users, far from being able to look for an alternative, won't even be aware of the problem.
Would it be possible for +18 tweets to be included without content (only metadata), even if the instance does not use an account?
That's a great find! The only thing that seems to be missing there is the pinned tweet.
And threads and retweets unfortunately also don't show up properly in the search
Unrelated to the last few posts: It's probably also worth noting that NSFW users don't show up in the search
Search uses a non-GraphQL endpoint, hence why it isn't affected in the same way. I've considered using search for timelines before, but it provides a much worse experience and is less reliable. I'll look into including sensitive tweets in timelines/RSS instead of omitting them.
I looked into this issue to understand it better, and it seems there's no way around it when using the GraphQL APIs, except authenticating with an account (which supposedly stays working even after the account gets suspended).
The global search doesn't show nsfw results. That didn't even work when I tried doing it with my usual authentication headers for some reason. When you access a profile directly, it should work tho (unless one of my instances / accounts (signified by the number in the parenthesis) is currently rate limited).
Just open https://nitter.privacydev.net/NAME_OF_NSFW_USER
Twitter blocks certain searches, including that hashtag. You can check by doing it on Twitter and checking the adaptive.json response in devtools, which should contain an error about the query being invalid.
For those searches to work, the account owner needs to go to https://twitter.com/settings/search and uncheck "Hide sensitive content".
Thank you! Fixed it.
@PrivacyDevel could you link to your fork or give any info on the changes made allowing for authentication? thanks!
My fork is here: https://github.com/PrivacyDevel/nitter
It's regular Nitter with the commits of the following PRs applied:
I have currently 4 instances. 3 instances with their own accounts each and a backup instance without any account that's used when all other instances have been rate limited. The selection is done by nginx, my load balancer.
On top of that I also set up caching + rate limiting with nginx as well, in an attempt to speed things up and get rate limited less frequently.
@PrivacyDevel thanks a lot, so I added the necessary variables to nitter.conf from a logged account, tested successful request with postman and ran it with docker compose, the instance doesn't do anything different to nitter.net. I don't get a responses as if logged. Am I missing something? it seems to just proxy to nitter.net
@PrivacyDevel Thanks for your work and for running the instance. Not sure if you're aware, but settings are not saved and sometimes pages have a different theme, probably because of the caching you're doing. Could be a good idea to stop access to the settings page.
I'm not entirely sure what you mean. If you have your own instance, it should be completely independent of nitter.net. And well, you can test if it worked by opening the profile of a user that posts nsfw content. If you see that kind of content, then it's working. If it does not show up it isn't working. Or you can check a profile page and check if you see a "like" tab and if it works. If it does, you're all set.
Note: if you want the search function to show nsfw results, you need to enable that here https://github.com/zedeus/nitter/issues/829#issuecomment-1510067533
Yeah, I was kind of afraid of this happening... Blocking the settings page won't be enough. I would have to ignore the cookies as well at the very least. But I don't really want to block off settings either... It's tricky.
Probably a dumb idea, but... would it be possible to add the caching layer between Twitter and Nitter? It would still reduce requests to Twitter, but Nitter itself would work normally (with delayed updates for popular content, I guess),
I guess I'm not understanding how to run the instance. I've compiled the code successfully, running a redis server but when I start ./nitter I get "Starting Nitter at http://nitter.net"
which is the hostname in the conf file. what am I missing?
That or using redis for caching more extensively would probably be the cleaner solution than what I am currently doing tbh. But it's also quite a bit more work. But I found a solution that should make it so that the issue you just described isn't going to happen anymore once the current cache expires.
You have to replace nitter.net with your own address.
@PrivacyDevel I did that but my instance looks just like nitter.net, shouldn't it have "nitter (NSFW 2)" on the upper left if I ran your fork?
No, that's just because I changed the title in my config file so that I can easily see which of my instances has been picked by the load balancer.
On first glance it should just look like normal Nitter. Because it mostly is.
great ! thanks a lot for your work, still have a hard time reading nim, wouldn’t have been able to add auth.
Thanks for your work! I'm running your fork locally for my own app that combines multiple twitter users into a single rss feeds. So I essentially have my own curated twitter home page without needing to use Twitter anymore.
That said, any plans to add this pull request as well? https://github.com/zedeus/nitter/pull/812
It fixes the search being broken for non-logged in accounts.
@PrivacyDevel Tried using your fork, but I'm trying to use it via Docker Compose. and I assume that when I do
docker compose up -dit pulls the files from zedeus version which have not been updated with this commit.Tried 3 different logged on twitter CT0/authtoken/xcsfrtoken combos but none of them work for me, still getting that I need to be logged in.
I could be wrong in this, but I am not sure what is causing the issue.
I've seen that at least some threads are not working properly with the GraphQL update anymore so I won't merge it into my fork just yet.
Yes, the docker-compose pulls the official docker image. You have to build it yourself.
Ok, well I'm not that well versed in how to do that, so I'll wait for zedeus to hopefully add the pull request. Or if I get the time to learn how to do it at a later time.
Thanks for the info though!
Managed to build it and make it work, however I had to edit this command
docker run -v $(pwd)/nitter.conf:/src/nitter.conf -d --network host nitter:latesttodocker run -v '$(pwd)'/nitter.conf -d --network host nitter:latestRunning the command from the guide i only got this error: docker: invalid reference format.
Now the "search" workaround doesn't even work anymore...
What do you mean? It works fine.
I meant from user specific searches like so:
https://nitter.net/[18+_twitter_handle]/search
This used to work as a workaround that jve-engineering found earlier but, a few days ago, even that stopped working. I've tried several instances as well with no luck.
Edit: It works again for some accounts, but not others. I don't really know what to say...
@PrivacyDevel Your workaround doesn't work if the NSFW twitter user is also blocked by the nitter host country. Unfortunately Germany censors things they do not like liberally, much more than other countries.
@orbea really? That sucks... Could you give me an example somehow by any chance?
@PrivacyDevel Certainly not the only account I have seen blocked by Germany, but the example I noticed now that prompted my comment.
While other nitter instances show the expected.
Hm.. that's really annoying..
Twitter in general and its recent API is really annoying right now and it seems to just keep getting worse every other week.
I try to tell people to not use proprietary social media platforms that can easily spy on, manipulate and censor their users, but it often falls on deaf ears... Even Minds depends on cloudflare.
Can we get a hub.docker.com image?
I read this discussion, but I still don't understand where to put the "session" information ?
I check to see if the option is in "nitter.conf", but it seems not.
Maybe it's simple, but I don't get the point.
Are there any pubic instances yet that show NSFW twitter posts?
cuz twitter (nitter) without NSFW is just as useless as tumblr without it
@SMylk I am thinking about it.
@c-chell it's probably easiest if you just use my fork for the time being.
@Owyn other than mine there are currently no other public ones afaik.
Thank you, I see you add a PR, I think will wait to get it to the main line.
In the discussion I understand that is already in it.
(I'm not really at ease to get around the Docker build)
I would like to request for this option to be added.
This should be only an optional setting that would be used to avoid getting rate limited by Twitter (when using the default account) and for also when the fork/mantainer doesn't include their own account.
I personally already have a Twitter account which I use to follow artists but I use nitter to open their feed without having to use lots of CPU such as when using the normal Twitter page. Having an optional setting would allow me to log in and get the content that I want without killing my PC.
@PrivacyDevel I found some tweets were still age locked even when using the NSFW3 instance. Interestingly, only some of these tweets are locked but others are properly shown
https://nitter.privacydev.net/huzeu1
Thank you for bringing this to my attention. I've never seen this before and was not able to reproduce it either. I feel like I know why this is happening tho. I guess the NFSW instances reused the data from the redis cache that is shared between all of the instances.
Tweets aren't cached, only pinned tweets, and they are never reused as part of the timeline if they show up again.
Hm ok. Thank you for this info @zedeus. Then I actually don't know what might've caused this..
What about when it shows "This tweet is unavailable"? is it the rate limit? cuz sometimes I get an RSS with a tweet (which is shown fine there), but when I open it on nitter (privacydev) it says "unavailable", but on twitter itself it shows fine If I try
@Owyn you'd have to provide an example for me to know
is it possible to add support for multiple accounts to get around the rate-limit?
Is there anything preventing #830 or #897 being merged to resolve this? I would like to stand up my own instance but I would like to view sensitive feeds.
Fixed.
Checked a dozen instances. Not actually fixed.
Depends on the country the instance runs in, and potentially the country from which its guest accounts were made, haven't tested that.
Can confirm that generating from the Netherlands and/or Finland (I mixed them, cant know which one worked) works with nsfw accounts
Hi,
I'm experiencing this issue on xcancel.com
Thanks