Replace tokens with guest accounts, swap endpoints #985

2023-08-18T18:37:42-04:00

zedeus commented

2023-08-18 18:37:42 -04:00

(Migrated from github.com)

This replaces the old method of using guest tokens with using "guest accounts".
For more information see this comment https://github.com/zedeus/nitter/issues/983#issuecomment-1683134811

Live now on nitter.net for testing. If you'd like to test and don't have any guest accounts, feel free to reach out to me on Matrix. A way to acquire guest accounts will be build and shared very soon.

This replaces the old method of using guest tokens with using "guest accounts". For more information see this comment https://github.com/zedeus/nitter/issues/983#issuecomment-1683134811 Live now on nitter.net for testing. If you'd like to test and don't have any guest accounts, feel free to reach out to me on Matrix. A way to acquire guest accounts will be build and shared very soon.

👀 1 🎉 4 🚀 2

dawnerd commented

2023-08-18 18:51:50 -04:00

(Migrated from github.com)

RSS returns forbidden, was this disabled?

veilm commented

2023-08-18 18:55:00 -04:00

(Migrated from github.com)

RSS returns forbidden, was this disabled?

https://github.com/zedeus/nitter/issues/937

> RSS returns forbidden, was this disabled? https://github.com/zedeus/nitter/issues/937

👍 1

DrSocket commented

2023-08-21 05:17:55 -04:00

(Migrated from github.com)

hi @zedeus, this is amazing, just a question, how is guest_accounts.json formatted?

zedeus commented

2023-08-21 05:21:27 -04:00

(Migrated from github.com)

It's a JSON array of the raw object you get from guest account flow, looks like this (you can use this one for testing if you want):

[{"user":{"id":1693553468835831800,"id_str":"1693553468835831808","name":"Open App User","screen_name":"_LO_082193WA35n","user_type":"Soft"},"next_link":{"link_type":"subtask","link_id":"next_link","subtask_id":"OpenAppFlowStartAccountSetupOpenLink"},"oauth_token":"1693553468835831808-TBis5qLp17jenKVyu9jeRjnTcTKjpL","oauth_token_secret":"BvbmNeJucYaUAWoEzrSUzTjkWxifoYaqAnjXTevL6ICxU","attribution_event":"signup"}]

It's a JSON array of the raw object you get from guest account flow, looks like this (you can use this one for testing if you want): ```json [{"user":{"id":1693553468835831800,"id_str":"1693553468835831808","name":"Open App User","screen_name":"_LO_082193WA35n","user_type":"Soft"},"next_link":{"link_type":"subtask","link_id":"next_link","subtask_id":"OpenAppFlowStartAccountSetupOpenLink"},"oauth_token":"1693553468835831808-TBis5qLp17jenKVyu9jeRjnTcTKjpL","oauth_token_secret":"BvbmNeJucYaUAWoEzrSUzTjkWxifoYaqAnjXTevL6ICxU","attribution_event":"signup"}] ```

👍 1 ❤️ 1

DrSocket commented

2023-08-21 05:26:31 -04:00

(Migrated from github.com)

Great! thanks a lot, it's the format from BANKA's script in an array.

👍 1

DrSocket commented

2023-08-21 06:52:12 -04:00

(Migrated from github.com)

I'm getting an error on compilation with this branch though, do you know anything about it?
parserutils.nim(39, 13) Error: undeclared field: 'notNull' for type packedjson.JsonNode [type declared in /Users/user/.nimble/pkgs/packedjson-#9e6fbb6/packedjson.nim(115, 3)]

I'm getting an error on compilation with this branch though, do you know anything about it? `parserutils.nim(39, 13) Error: undeclared field: 'notNull' for type packedjson.JsonNode [type declared in /Users/user/.nimble/pkgs/packedjson-#9e6fbb6/packedjson.nim(115, 3)]`

zedeus commented

2023-08-21 07:17:44 -04:00

(Migrated from github.com)

This happens with Nim versions older than 1.6.14, I'll fix it soon, for now you can work around it by updating to 1.6.14 or preferably 2.0

pbregener commented

2023-08-21 07:47:12 -04:00

(Migrated from github.com)

For the record: I successfully built a Docker image for aarch64 from this branch and it works like a charm :)

I changed the base image to use alpine 3.18 which has Nim 1.6.14

For the record: I successfully built a Docker image for aarch64 from this branch and it works like a charm :) I changed the base image to use alpine 3.18 which has Nim 1.6.14

DrSocket commented

2023-08-21 08:18:10 -04:00

(Migrated from github.com)

Sorry for the long post, couple of questions just for my own learning:

Do you using Android Studio to reverse engineer the android endpoints? If not how do you recommend going about it?
Would it be complicated to add an option to use account credentials from a json instead of guest account in case they close the guest accounts endpoint and just have a lot of them. I'm still pretty rusty with nim code, maybe you have recommendations about a good place so start learning so I can actually help contributing to the project (currently use C,C++,python,TS).
i.e. (this account is blocked now) ->
{"cookie":"_twitter_sess=xyz; kdt=xyz; auth_token=xyz; ct0=xyz; personalization_id="xyz"; twid="xyz, "x-csrf-token": "xyz", "authorization": "Bearer AAAAAAAAAAAAAAAAAAAAAxyz", "x-twitter-auth-type": "OAuth2Session", "x-twitter-active-user": "yes"}

Sorry for the long post, couple of questions just for my own learning: 1. Do you using Android Studio to reverse engineer the android endpoints? If not how do you recommend going about it? 2. Would it be complicated to add an option to use account credentials from a json instead of guest account in case they close the guest accounts endpoint and just have a lot of them. I'm still pretty rusty with nim code, maybe you have recommendations about a good place so start learning so I can actually help contributing to the project (currently use C,C++,python,TS). i.e. (this account is blocked now) -> {"cookie":"_twitter_sess=xyz; kdt=xyz; auth_token=xyz; ct0=xyz; personalization_id="xyz"; twid="xyz, "x-csrf-token": "xyz", "authorization": "Bearer AAAAAAAAAAAAAAAAAAAAAxyz", "x-twitter-auth-type": "OAuth2Session", "x-twitter-active-user": "yes"}

zedeus commented

2023-08-21 08:52:02 -04:00

(Migrated from github.com)

I used a virtual machine to listen to network requests, but most of the reverse engineering work was done by other folks.

use account credentials from a json instead of guest account in case they close the guest accounts endpoint

I'm not sure what you mean since that's how it already works, storing guest accounts in a file. No endpoints are being used beyond the initial creation of the account, which expires after 30 days. When you say blocked, do you mean you get an error code 88 (rate limit exceeded)? That block goes away 24 hours later.

I used a virtual machine to listen to network requests, but most of the reverse engineering work was done by other folks. > use account credentials from a json instead of guest account in case they close the guest accounts endpoint I'm not sure what you mean since that's how it already works, storing guest accounts in a file. No endpoints are being used beyond the initial creation of the account, which expires after 30 days. When you say blocked, do you mean you get an error code 88 (rate limit exceeded)? That block goes away 24 hours later.

DrSocket commented

2023-08-21 09:00:47 -04:00

(Migrated from github.com)

Would you mind sharing what the kind of vm you use for this? something like wireshark or mitmproxy?
That's an actual burner account, not a guest account. It would be great if we could switch from guest to burner accounts like PrivacyDevel uses. In case it gets shut down again just have a few burners to still be able to use Nitter while finding a new solution.
By locked I mean:
"Twitter responded with an error: AuthorizationError: Authorization: Denied by access control: To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to https://twitter.com to unlock your account."

Would you mind sharing what the kind of vm you use for this? something like wireshark or mitmproxy? That's an actual burner account, not a guest account. It would be great if we could switch from guest to burner accounts like PrivacyDevel uses. In case it gets shut down again just have a few burners to still be able to use Nitter while finding a new solution. By locked I mean: "Twitter responded with an error: AuthorizationError: Authorization: Denied by access control: To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to https://twitter.com to unlock your account."

zedeus commented

2023-08-21 09:07:40 -04:00

(Migrated from github.com)

I used the free trial from Corellium to run Twitter and inspect network logs. I tried running BlissOS via qemu to no avail first.

I just noticed the thing you posted is a normal account, didn't look close enough. What I said still stands for guest accounts, they don't get blocked though just limited for 24 hours. The problem with burner accounts is you need a lot for any meaningfully big instance, so it's not really feasible especially since they just get locked. It's extremely easy to get guest accounts, just takes a lot of proxies which are easy to get.

I used the free trial from Corellium to run Twitter and inspect network logs. I tried running BlissOS via qemu to no avail first. I just noticed the thing you posted is a normal account, didn't look close enough. What I said still stands for guest accounts, they don't get _blocked_ though just limited for 24 hours. The problem with burner accounts is you need a _lot_ for any meaningfully big instance, so it's not really feasible especially since they just get locked. It's extremely easy to get guest accounts, just takes a lot of proxies which are easy to get.

DrSocket commented

2023-08-21 09:14:45 -04:00

(Migrated from github.com)

Great thank you!
Yes the problem with guest accounts solution is that twitter can still kill this method no?

Great thank you! Yes the problem with guest accounts solution is that twitter can still kill this method no?

zedeus commented

2023-08-21 09:25:34 -04:00

(Migrated from github.com)

Indeed, but that's the case with every solution. With accounts they can start banning or further restricting, there is no perfect solution.

felipefalkiner commented

2023-08-21 15:00:29 -04:00

(Migrated from github.com)

How to upgrade to this version?
Git clone this branch and run the following commands again?

$ nimble build -d:release
$ nimble scss
$ nimble md

How to upgrade to this version? Git clone this branch and run the following commands again? ``` $ nimble build -d:release $ nimble scss $ nimble md ```

zedeus commented

2023-08-21 15:59:27 -04:00

(Migrated from github.com)

Correct, but you also need to get one or more guest accounts, otherwise nothing will work

dawnerd commented

2023-08-21 18:20:42 -04:00

(Migrated from github.com)

I have this running on my instance and so far it's running great, haven't seen any issues so far. Getting gust tokens definitely the trickiest part, but being as Im just running a private/relay instance one guest user should cover it.

Thanks for getting this put together!

I have this running on my instance and so far it's running great, haven't seen any issues so far. Getting gust tokens definitely the trickiest part, but being as Im just running a private/relay instance one guest user should cover it. Thanks for getting this put together!

somini commented

2023-08-21 19:43:08 -04:00

(Migrated from github.com)

Is it possible to run this branch with an actual account? What are the tokens I need to extract from the real account to create the JSON file?

zedeus commented

2023-08-21 19:53:36 -04:00

(Migrated from github.com)

It might be possible but I haven't tried, there's no code in this branch for regular accounts as they use cookies and bearer tokens etc instead of oauth.

cmj commented

2023-08-21 19:59:34 -04:00

(Migrated from github.com)

Do you happen to know the rate limits for with_replies? I am burning through the limit fast with replies but the timelines seems stable for my private instances. Overall this is working great.

zedeus commented

2023-08-21 20:06:58 -04:00

(Migrated from github.com)

I know what the rate limit is supposed to be, but what it actually is I don't know. The same question was just asked here: https://github.com/zedeus/nitter/issues/983#issuecomment-1687182471

cmj commented

2023-08-21 20:10:18 -04:00

(Migrated from github.com)

Thanks. I'll keep it on timelines for now until I can get a proxy system setup.

felipefalkiner commented

2023-08-22 09:09:48 -04:00

(Migrated from github.com)

There's any guide on how to get guest accounts?

> There's any guide on how to get guest accounts?

zedeus commented

2023-08-22 09:45:47 -04:00

(Migrated from github.com)

@felipefalkiner see here: https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357
TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to console.log(JSON.stringify(account)), then you can put it in a JSON array (looks like [{"user": ...}]) in the guest_accounts.json file Nitter expects in the directory you run it from.

@felipefalkiner see here: https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357 TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to `console.log(JSON.stringify(account))`, then you can put it in a JSON array (looks like `[{"user": ...}]`) in the guest_accounts.json file Nitter expects in the directory you run it from.

❤️ 2

unixfox commented

2023-08-23 11:54:24 -04:00

(Migrated from github.com)

For local/private instance, I propose to have nitter automatically create guest accounts and store it somewhere in the disk.

I think it should be enough for local usage to have at least just one guest account.

And for bigger instance (public ones), the owner can provide a guest accounts file, like already implemented in this PR.

For local/private instance, I propose to have nitter automatically create guest accounts and store it somewhere in the disk. I think it should be enough for local usage to have at least just one guest account. And for bigger instance (public ones), the owner can provide a guest accounts file, like already implemented in this PR.

zedeus commented

2023-08-23 13:20:59 -04:00

(Migrated from github.com)

That's the plan, except I'll host a guest account pool service that public instances (those in the wiki) can automatically connect to, so they don't need to worry about proxies or anything.

That's the plan, except I'll host a guest account pool service that _public_ instances (those in the wiki) can automatically connect to, so they don't need to worry about proxies or anything.

somini commented

2023-08-23 19:09:12 -04:00

(Migrated from github.com)

@felipefalkiner see here: #983 (comment) TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to console.log(JSON.stringify(account)), then you can put it in a JSON array (looks like [{"user": ...}]) in the guest_accounts.json file Nitter expects in the directory you run it from.

Thanks for that.

In case anyone know nothing about node:

Create an empty directory
Create a file guest.js and paste that script's content (including JSON.stringify change))
Create a file package.json with the extact contents: {"type":"module"}
cd to that directory and run node guest

That will create the JSON object. Adapt it to a JSON array (maybe run the script from several IPs?), and add it to guest_accounts.json on nitter's working directory.

> @felipefalkiner see here: [#983 (comment)](https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357) TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to `console.log(JSON.stringify(account))`, then you can put it in a JSON array (looks like `[{"user": ...}]`) in the guest_accounts.json file Nitter expects in the directory you run it from. Thanks for that. In case anyone know nothing about node: - Create an empty directory - Create a file `guest.js` and paste that script's content (including `JSON.stringify` change)) - Create a file `package.json` with the extact contents: `{"type":"module"}` - `cd` to that directory and run `node guest` That will create the JSON object. Adapt it to a JSON array (maybe run the script from several IPs?), and add it to `guest_accounts.json` on nitter's working directory.

👍 4

cmj commented

2023-08-23 19:27:53 -04:00

(Migrated from github.com)

Adding onto this, I know next to nothing about node, but was able to get this working passing the proxy host:port to the script. In theory it should work but all proxies I tested today didn't work:
https://gist.github.com/cmj/79501b374e387aebe3a717b9e8edda04

And because node seems to still be foreign to me I wanted to break it out into a simple shell script:
https://gist.github.com/cmj/6e6f6cae51c28cf6e161ceba8d108dda

Here is the same but with using proxy option:
https://gist.github.com/cmj/6cb90eae1ed299ee050a2396e7ceaaaa

Adding onto this, I know next to nothing about node, but was able to get this working passing the proxy host:port to the script. In theory it should work but all proxies I tested today didn't work: https://gist.github.com/cmj/79501b374e387aebe3a717b9e8edda04 And because node seems to still be foreign to me I wanted to break it out into a simple shell script: https://gist.github.com/cmj/6e6f6cae51c28cf6e161ceba8d108dda Here is the same but with using proxy option: https://gist.github.com/cmj/6cb90eae1ed299ee050a2396e7ceaaaa

❤️ 6

somini commented

2023-08-24 17:35:28 -04:00

(Migrated from github.com)

@cmj Thanks, that's much easier. That script should be included on the repository itself.

mhkhung commented

2023-08-29 09:58:30 -04:00

(Migrated from github.com)

I have been running this branch with a few tokens for a few days already.. seems to be working well other than hitting "Instance has been rate limited." when viewing individual tweets.. is having more tokens going to solve this problem? Or are they rate-limiting my server IP?

If each token's going to last about a month, we probably should tag them with a generation date so we can rotate them out easily/automatically.

I have been running this branch with a few tokens for a few days already.. seems to be working well other than hitting "Instance has been rate limited." when viewing individual tweets.. is having more tokens going to solve this problem? Or are they rate-limiting my server IP? If each token's going to last about a month, we probably should tag them with a generation date so we can rotate them out easily/automatically.

👍 2

zedeus commented

2023-08-29 16:15:10 -04:00

(Migrated from github.com)

@mhkhung Yes, more accounts will help (they're not called tokens to avoid confusion). Keeping track of accounts and handling them better is on the todo list. This is all very much WIP.

❤️ 1

redfalcoon commented

2023-08-31 10:57:02 -04:00

(Migrated from github.com)

To avoid the overload of public instances, I've built the last version committed ( Version [2023.08.31-84dcf49]).
After few days of usage of then branch with guest accounts, and the progressives builts, it's seems that everythings works well even if from time to time I got the message "Instance has been rate limited. Use [another instance]".
Now running this new configuration, I had few questions about guest accounts.
The right knowledge of the correct approach to guest accounts, based on yours knowledge and experiences, can be usefull to address accordingly further and future issues (not related to Musk X paturnies).
The questions that I have are:
- For the local instance there is a rule of thumb about the number of the guest accounts?
- There is a way to check what his the guest account in use by Nitter application in a running environment, to keep track of
the usage and of the rolling management of it?
- I've seen that the screen name of the guest account (generated using procedures like those created by @cmj) has the prefix
_LO_mmdd (with mm that specify the month and dd that specify the date of the creation date of the account),
those information are used to expire the account? If this is true the account generation process must be structured
accordingly, expiring the accounts after their issue time limit.
Thanks to all for the various usefull comments and to @zedeus for his great work.

To avoid the overload of public instances, I've built the last version committed ( Version [2023.08.31-84dcf49]). After few days of usage of then branch with guest accounts, and the progressives builts, it's seems that everythings works well even if from time to time I got the message "Instance has been rate limited. Use [another instance]". Now running this new configuration, I had few questions about guest accounts. The right knowledge of the correct approach to guest accounts, based on yours knowledge and experiences, can be usefull to address accordingly further and future issues (not related to Musk X paturnies). The questions that I have are: - For the local instance there is a rule of thumb about the number of the guest accounts? - There is a way to check what his the guest account in use by Nitter application in a running environment, to keep track of the usage and of the rolling management of it? - I've seen that the screen name of the guest account (generated using procedures like those created by @cmj) has the prefix _LO_mmdd (with mm that specify the month and dd that specify the date of the creation date of the account), those information are used to expire the account? If this is true the account generation process must be structured accordingly, expiring the accounts after their issue time limit. Thanks to all for the various usefull comments and to @zedeus for his great work.

AlexGuo1998 commented

2023-09-04 12:57:34 -04:00

(Migrated from github.com)

Hi. Just deployed guest_accounts branch to my server (with 2 guest accounts). However, it seems to be running very slow.
When I open a user's timeline, it takes several minutes for the pictures to fully load.

The log shows nothing. I forwarded the port directly to nitter, so this should not be a proxy issue.
The debug endpoint (/.tokens) is also blocked when loading the images.

Do you have any ideas? Could it be a locking issue?

Hi. Just deployed `guest_accounts` branch to my server (with 2 guest accounts). However, it seems to be running very slow. When I open a user's timeline, it takes several minutes for the pictures to fully load. ![Firefox developer tools, network page](https://github.com/zedeus/nitter/assets/25975864/0361900c-ac28-494a-8c14-6f1094087712) The log shows nothing. I forwarded the port directly to nitter, so this should not be a proxy issue. The debug endpoint (`/.tokens`) is also blocked when loading the images. Do you have any ideas? Could it be a locking issue?

zedeus commented

2023-09-04 19:34:42 -04:00

(Migrated from github.com)

No clue, if you're using Nginx it might be weird buffering behaviour or a low file/socket limit. Try adding this to the location rule:

proxy_buffers 64 16k;  
proxy_buffer_size 4k;

No clue, if you're using Nginx it might be weird buffering behaviour or a low file/socket limit. Try adding this to the location rule: ``` proxy_buffers 64 16k; proxy_buffer_size 4k; ```

AlexGuo1998 commented

2023-09-04 23:21:43 -04:00

(Migrated from github.com)

Update: I'm not going to create an issue just for this.
Eventually I found out there's something wrong with my DNS setup. The name resolution sometimes hang for 5+ seconds, and blocked the whole thread (and other requests).

Old comment

I didn't use Nginx in the previous example. Instead, I SSHed into my server and set up a port forward, to allow Firefox to talk to Nitter directly.
I also tried using Nginx (with and without the above rules), but it is still very slow.

Just noticed that there are several timeouts in the logs:

error: ProtocolError, msg: Connection was closed before full request has been made, accountId: <...>, url: https://api.twitter.com/graphql/3JNH4e9dq1BifLxAa3UMWg/UserWithProfileTweetsQueryV2?variables=<...>
[accounts] Rate limited, retrying userTweets request...
error: ProtocolError, msg: Connection was closed before full request has been made, accountId: <...>, url: https://api.twitter.com/graphql/3JNH4e9dq1BifLxAa3UMWg/UserWithProfileTweetsQueryV2?variables=<...>
[accounts] Rate limited, retrying userTweets request...
[media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb
[media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb
[media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb
[media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb
[media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb

Does this indicate that my server is rate-limited by Twitter (possibly at the socket level)?
This instance is accessible only by myself, and I use it only for normal browsing. It shouldn't hit the rate limit.

I'll try to capture and inspect some tcpdumps later.

Update: I'm not going to create an issue just for this. Eventually I found out there's something wrong with my DNS setup. The name resolution sometimes hang for 5+ seconds, and blocked the whole thread (and other requests). <details> <summary>Old comment</summary> I didn't use Nginx in the previous example. Instead, I SSHed into my server and set up a port forward, to allow Firefox to talk to Nitter directly. I also tried using Nginx (with and without the above rules), but it is still very slow. Just noticed that there are several timeouts in the logs: ```text error: ProtocolError, msg: Connection was closed before full request has been made, accountId: <...>, url: https://api.twitter.com/graphql/3JNH4e9dq1BifLxAa3UMWg/UserWithProfileTweetsQueryV2?variables=<...> [accounts] Rate limited, retrying userTweets request... error: ProtocolError, msg: Connection was closed before full request has been made, accountId: <...>, url: https://api.twitter.com/graphql/3JNH4e9dq1BifLxAa3UMWg/UserWithProfileTweetsQueryV2?variables=<...> [accounts] Rate limited, retrying userTweets request... [media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb [media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb [media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb [media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb [media] Proxying exception, error: Connection was closed before full request has been made, url: https://pbs.twimg.com/media/<...>.jpg:thumb ``` Does this indicate that my server is rate-limited by Twitter (possibly at the socket level)? This instance is accessible only by myself, and I use it only for normal browsing. It shouldn't hit the rate limit. I'll try to capture and inspect some tcpdumps later. </details>

unixfox commented

2023-09-05 03:44:53 -04:00

(Migrated from github.com)

That's 100% an issue with your setup. It works fine for everyone.

Could you create a new github issue for discussing that? this PR is already filled with many comments so discussing an unrelated problem will make things much worse.

That's 100% an issue with your setup. It works fine for everyone. Could you create a new github issue for discussing that? this PR is already filled with many comments so discussing an unrelated problem will make things much worse.

👍 2

Lomanic (Migrated from github.com) reviewed

2023-10-31 07:09:57 -04:00

nitter.nimble

					
				@ -16,3 +16,3 @@

				requires "sass#7dfdd03"

				requires "nimcrypto#4014ef9"

				requires "nimcrypto#a079df9"

				requires "markdown#158efe3"

this bumps your requires nim to requires "nim >= 1.6", nitter doesn't build anymore with nim 1.4.8 because of this.

this bumps your `requires nim` to [`requires "nim >= 1.6"`](https://github.com/cheatfate/nimcrypto/blob/a079df92424968d46a6ac258299ce9380aa153f2/nimcrypto.nimble#L11), nitter doesn't build anymore with nim 1.4.8 because of this.

zedeus (Migrated from github.com) reviewed

2023-10-31 07:29:55 -04:00

nitter.nimble

					
				@ -16,3 +16,3 @@

				requires "sass#7dfdd03"

				requires "nimcrypto#4014ef9"

				requires "nimcrypto#a079df9"

				requires "markdown#158efe3"

Nice catch, thanks. 1.6.0 was released two full years ago, so I think it's time to bump that anyway. Debian is on 1.6.14 and Ubuntu 23.04 is 1.6.10, so I'll set that as the minimum.

unixfox (Migrated from github.com) reviewed

2023-10-31 08:47:20 -04:00

nitter.nimble

					
				@ -16,3 +16,3 @@

				requires "sass#7dfdd03"

				requires "nimcrypto#4014ef9"

				requires "nimcrypto#a079df9"

				requires "markdown#158efe3"

I have created a PR for that: https://github.com/zedeus/nitter/pull/1053

Lomanic (Migrated from github.com) reviewed

2023-10-31 11:34:51 -04:00

nitter.nimble

					
				@ -16,3 +16,3 @@

				requires "sass#7dfdd03"

				requires "nimcrypto#4014ef9"

				requires "nimcrypto#a079df9"

				requires "markdown#158efe3"

Ideally your CI should build nitter with this minimum version instead of some loose 1.x to catch this in the future.

Ideally your CI should build nitter with this minimum version [instead of some loose 1.x](https://github.com/zedeus/nitter/blob/d7ca353a55ea3440a2ec1f09155951210a374cc7/.github/workflows/run-tests.yml#L31) to catch this in the future.

unixfox commented

2023-10-31 19:28:18 -04:00

(Migrated from github.com)

Could you rebase against master in order to include this commit: github.com/zedeus/nitter@b62d73dbd3?

Thank you

Could you rebase against master in order to include this commit: https://github.com/zedeus/nitter/commit/b62d73dbd373f08af07c7a79efcd790d3bc1a49c? Thank you

👍 1

somini commented

2023-11-14 21:08:45 -05:00

(Migrated from github.com)

What's the status of this? Can nitter itself at least run the guest script itself once a day to replenish the tokens?

👍 3

y0nei commented

2023-12-15 09:54:01 -05:00

(Migrated from github.com)

I hacked myself a small script to auto generate guest accounts with the guest_tokens.sh script and (properly) append created entries to a guest_accounts.json file between [ and ] characters.
guest_tokens.sh:

#!/bin/bash
# Grab Twitter guest account tokens for use with Nitter.

guest_token=$(curl -s -XPOST https://api.twitter.com/1.1/guest/activate.json -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' | jq -r '.guest_token')

flow_token=$(curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json?flow_name=welcome&api_version=1&known_device_token=&sim_country_code=us' \
  -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \
  -H 'X-Twitter-API-Version: 5' \
  -H 'X-Twitter-Client: TwitterAndroid' \
  -H 'X-Twitter-Client-Version: 9.95.0-release.0' \
  -H 'OS-Version: 28' \
  -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \
  -H 'X-Twitter-Active-User: yes' \
  -H "X-Guest-Token: ${guest_token}" \
  -d '{"flow_token":null,"input_flow_data":{"country_code":null,"flow_context":{"start_location":{"location":"splash_screen"}},"requested_variant":null,"target_user_id":0},"subtask_versions":{"generic_urt":3,"standard":1,"open_home_timeline":1,"app_locale_update":1,"enter_date":1,"email_verification":3,"enter_password":5,"enter_text":5,"one_tap":2,"cta":7,"single_sign_on":1,"fetch_persisted_data":1,"enter_username":3,"web_modal":2,"fetch_temporary_password":1,"menu_dialog":1,"sign_up_review":5,"interest_picker":4,"user_recommendations_urt":3,"in_app_notification":1,"sign_up":2,"typeahead_search":1,"user_recommendations_list":4,"cta_inline":1,"contacts_live_sync_permission_prompt":3,"choice_selection":5,"js_instrumentation":1,"alert_dialog_suppress_client_events":1,"privacy_options":1,"topics_selector":1,"wait_spinner":3,"tweet_selection_urt":1,"end_flow":1,"settings_list":7,"open_external_link":1,"phone_verification":5,"security_key":3,"select_banner":2,"upload_media":1,"web":2,"alert_dialog":1,"open_account":2,"action_list":2,"enter_phone":2,"open_link":1,"show_code":1,"update_users":1,"check_logged_in_account":1,"enter_email":2,"select_avatar":4,"location_permission_prompt":2,"notifications_permission_prompt":4}}' | jq -r .flow_token)

curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json' \
  -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \
  -H 'X-Twitter-API-Version 5' \
  -H 'X-Twitter-Client: TwitterAndroid' \
  -H 'X-Twitter-Client-Version: 9.95.0-release.0' \
  -H 'OS-Version: 28' \
  -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \
  -H 'X-Twitter-Active-User: yes' \
  -H "X-Guest-Token: ${guest_token}" \
  -d "{\"flow_token\":\"${flow_token}\",\"subtask_inputs\":[{\"open_link\":{\"link\":\"next_link\"},\"subtask_id\":\"NextTaskOpenLink\"}],\"subtask_versions\":{\"generic_urt\":3,\"standard\":1,\"open_home_timeline\":1,\"app_locale_update\":1,\"enter_date\":1,\"email_verification\":3,\"enter_password\":5,\"enter_text\":5,\"one_tap\":2,\"cta\":7,\"single_sign_on\":1,\"fetch_persisted_data\":1,\"enter_username\":3,\"web_modal\":2,\"fetch_temporary_password\":1,\"menu_dialog\":1,\"sign_up_review\":5,\"interest_picker\":4,\"user_recommendations_urt\":3,\"in_app_notification\":1,\"sign_up\":2,\"typeahead_search\":1,\"user_recommendations_list\":4,\"cta_inline\":1,\"contacts_live_sync_permission_prompt\":3,\"choice_selection\":5,\"js_instrumentation\":1,\"alert_dialog_suppress_client_events\":1,\"privacy_options\":1,\"topics_selector\":1,\"wait_spinner\":3,\"tweet_selection_urt\":1,\"end_flow\":1,\"settings_list\":7,\"open_external_link\":1,\"phone_verification\":5,\"security_key\":3,\"select_banner\":2,\"upload_media\":1,\"web\":2,\"alert_dialog\":1,\"open_account\":2,\"action_list\":2,\"enter_phone\":2,\"open_link\":1,\"show_code\":1,\"update_users\":1,\"check_logged_in_account\":1,\"enter_email\":2,\"select_avatar\":4,\"location_permission_prompt\":2,\"notifications_permission_prompt\":4}}" | jq -r '.subtasks[0]|if(.open_account) then .open_account else empty end'

generate_guest_accounts.sh:

#!/bin/bash

# Run script and capture its output
output=$(./guest_tokens.sh)

# Check if the output is not empty
if [ -n "$output" ]; then
  # Check if the JSON file exists
  if [ ! -f guest_accounts.json ]; then
    # If the file does not exist, create it with an opening square bracket
    echo "[" > guest_accounts.json
  else
    # If the file already exists, remove the trailing square bracket
    sed -i '$ d' guest_accounts.json
    # Add a comma between entries
    echo "," >> guest_accounts.json
  fi

  # Append the output to the JSON file and add closing square bracket
  echo $output >> guest_accounts.json
  echo "]" >> guest_accounts.json

  # Format the JSON file using jq
  jq . guest_accounts.json > output-formatted.json
  mv output-formatted.json guest_accounts.json

  # Print a success message
  echo "Entry appended to and formatted in json file."
else
  echo "No output from script. Nothing appended to json file."
fi

Now generated entries should be appended while maintaining proper JSON syntax, I also format the whole file with jq for readability.
NOTE: jq, sed and curl are required to be installed on your system.

Finally you can use crontab -e to schedule execution of this script to your liking in order to fully automate the process of account generation:

# Every day at 12AM
0 0 * * * cd $HOME/Docker/nitter && ./generate_guest_accounts.sh

Here is a sample script used for testing in order to simulate what generate_tokens.sh would do:

#!/bin/bash

# Generate a JSON block with a random user ID
generate_json_entry() {
  user_id=$((RANDOM % 1000000))
  cat <<EOF
{
  "user": {
    "id": $user_id
  }
}
EOF
}

generate_json_entry

I hacked myself a small script to auto generate guest accounts with the `guest_tokens.sh` script and (properly) append created entries to a `guest_accounts.json` file between `[` and `]` characters. `guest_tokens.sh`: ```bash #!/bin/bash # Grab Twitter guest account tokens for use with Nitter. guest_token=$(curl -s -XPOST https://api.twitter.com/1.1/guest/activate.json -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' | jq -r '.guest_token') flow_token=$(curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json?flow_name=welcome&api_version=1&known_device_token=&sim_country_code=us' \ -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \ -H 'Content-Type: application/json' \ -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \ -H 'X-Twitter-API-Version: 5' \ -H 'X-Twitter-Client: TwitterAndroid' \ -H 'X-Twitter-Client-Version: 9.95.0-release.0' \ -H 'OS-Version: 28' \ -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \ -H 'X-Twitter-Active-User: yes' \ -H "X-Guest-Token: ${guest_token}" \ -d '{"flow_token":null,"input_flow_data":{"country_code":null,"flow_context":{"start_location":{"location":"splash_screen"}},"requested_variant":null,"target_user_id":0},"subtask_versions":{"generic_urt":3,"standard":1,"open_home_timeline":1,"app_locale_update":1,"enter_date":1,"email_verification":3,"enter_password":5,"enter_text":5,"one_tap":2,"cta":7,"single_sign_on":1,"fetch_persisted_data":1,"enter_username":3,"web_modal":2,"fetch_temporary_password":1,"menu_dialog":1,"sign_up_review":5,"interest_picker":4,"user_recommendations_urt":3,"in_app_notification":1,"sign_up":2,"typeahead_search":1,"user_recommendations_list":4,"cta_inline":1,"contacts_live_sync_permission_prompt":3,"choice_selection":5,"js_instrumentation":1,"alert_dialog_suppress_client_events":1,"privacy_options":1,"topics_selector":1,"wait_spinner":3,"tweet_selection_urt":1,"end_flow":1,"settings_list":7,"open_external_link":1,"phone_verification":5,"security_key":3,"select_banner":2,"upload_media":1,"web":2,"alert_dialog":1,"open_account":2,"action_list":2,"enter_phone":2,"open_link":1,"show_code":1,"update_users":1,"check_logged_in_account":1,"enter_email":2,"select_avatar":4,"location_permission_prompt":2,"notifications_permission_prompt":4}}' | jq -r .flow_token) curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json' \ -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \ -H 'Content-Type: application/json' \ -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \ -H 'X-Twitter-API-Version 5' \ -H 'X-Twitter-Client: TwitterAndroid' \ -H 'X-Twitter-Client-Version: 9.95.0-release.0' \ -H 'OS-Version: 28' \ -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \ -H 'X-Twitter-Active-User: yes' \ -H "X-Guest-Token: ${guest_token}" \ -d "{\"flow_token\":\"${flow_token}\",\"subtask_inputs\":[{\"open_link\":{\"link\":\"next_link\"},\"subtask_id\":\"NextTaskOpenLink\"}],\"subtask_versions\":{\"generic_urt\":3,\"standard\":1,\"open_home_timeline\":1,\"app_locale_update\":1,\"enter_date\":1,\"email_verification\":3,\"enter_password\":5,\"enter_text\":5,\"one_tap\":2,\"cta\":7,\"single_sign_on\":1,\"fetch_persisted_data\":1,\"enter_username\":3,\"web_modal\":2,\"fetch_temporary_password\":1,\"menu_dialog\":1,\"sign_up_review\":5,\"interest_picker\":4,\"user_recommendations_urt\":3,\"in_app_notification\":1,\"sign_up\":2,\"typeahead_search\":1,\"user_recommendations_list\":4,\"cta_inline\":1,\"contacts_live_sync_permission_prompt\":3,\"choice_selection\":5,\"js_instrumentation\":1,\"alert_dialog_suppress_client_events\":1,\"privacy_options\":1,\"topics_selector\":1,\"wait_spinner\":3,\"tweet_selection_urt\":1,\"end_flow\":1,\"settings_list\":7,\"open_external_link\":1,\"phone_verification\":5,\"security_key\":3,\"select_banner\":2,\"upload_media\":1,\"web\":2,\"alert_dialog\":1,\"open_account\":2,\"action_list\":2,\"enter_phone\":2,\"open_link\":1,\"show_code\":1,\"update_users\":1,\"check_logged_in_account\":1,\"enter_email\":2,\"select_avatar\":4,\"location_permission_prompt\":2,\"notifications_permission_prompt\":4}}" | jq -r '.subtasks[0]|if(.open_account) then .open_account else empty end' ``` `generate_guest_accounts.sh`: ```bash #!/bin/bash # Run script and capture its output output=$(./guest_tokens.sh) # Check if the output is not empty if [ -n "$output" ]; then # Check if the JSON file exists if [ ! -f guest_accounts.json ]; then # If the file does not exist, create it with an opening square bracket echo "[" > guest_accounts.json else # If the file already exists, remove the trailing square bracket sed -i '$ d' guest_accounts.json # Add a comma between entries echo "," >> guest_accounts.json fi # Append the output to the JSON file and add closing square bracket echo $output >> guest_accounts.json echo "]" >> guest_accounts.json # Format the JSON file using jq jq . guest_accounts.json > output-formatted.json mv output-formatted.json guest_accounts.json # Print a success message echo "Entry appended to and formatted in json file." else echo "No output from script. Nothing appended to json file." fi ``` Now generated entries should be appended while maintaining proper JSON syntax, I also format the whole file with `jq` for readability. `NOTE`: `jq`, `sed` and `curl` are required to be installed on your system. Finally you can use `crontab -e` to schedule execution of this script to your liking in order to fully automate the process of account generation: ```sh # Every day at 12AM 0 0 * * * cd $HOME/Docker/nitter && ./generate_guest_accounts.sh ``` *** Here is a sample script used for testing in order to simulate what `generate_tokens.sh` would do: ```bash #!/bin/bash # Generate a JSON block with a random user ID generate_json_entry() { user_id=$((RANDOM % 1000000)) cat <<EOF { "user": { "id": $user_id } } EOF } generate_json_entry ```

zedeus commented

2023-12-15 10:34:21 -05:00

(Migrated from github.com)

@y0nei JSONL is supported, you don't need to do all that. Just append each json object on a new line and name the file .jsonl instead of .json.

@y0nei JSONL is supported, you don't need to do all that. Just append each json object on a new line and name the file `.jsonl` instead of `.json`.

👍 3

somini commented

2024-01-25 17:31:22 -05:00

(Migrated from github.com)

It seems the script for getting guest accounts it broken now. And maybe they expired all those credentials, I only get what's already cached in my instance.

👍 3

y0nei commented

2024-01-26 06:20:47 -05:00

(Migrated from github.com)

It seems the script for getting guest accounts it broken now. And maybe they expired all those credentials, I only get what's already cached in my instance.

Same issue for me, I'm unable to generate any new guest accounts, all of my current ones got "expired" and I can only view what is cached on my instance.

> It seems the script for getting guest accounts it broken now. And maybe they expired all those credentials, I only get what's already cached in my instance. Same issue for me, I'm unable to generate any new guest accounts, all of my current ones got "expired" and I can only view what is cached on my instance.

nyanpasu64 commented

2024-01-27 04:22:17 -05:00

(Migrated from github.com)

Oddly I can view posts in Nitter again with my old tokens, but the guest token generation script still returns undefined (possibly it's been less than a day since my last failed guest token generation during the Nitter outage?). Let's see how long until Twitter kills less manipulative frontends again.

Even during the outage, I could view user profiles and old tweets, but expanding any tweet resulted in an error.

Oddly I can view posts in Nitter again with my old tokens, but the guest token generation script still returns `undefined` (possibly it's been less than a day since my last failed guest token generation during the Nitter outage?). Let's see how long until Twitter kills less manipulative frontends again. Even during the outage, I could view user profiles and old tweets, but expanding any tweet resulted in an error.

👍 1

somini commented

2024-01-27 18:19:30 -05:00

(Migrated from github.com)

Can confirm, the token script fails, but I can read Nitter again, mostly. Seems hit and miss, the RSS feeds show the tweet, but I can't see the tweet responses for example.

zedeus commented

2025-02-05 14:58:01 -05:00

(Migrated from github.com)

Replaced by #1210