Avatar
Guy Van Sanden
eb2881406ad19ba7cf01c210ce002f4fe53e8ce6d84e77df5a2319f9f00a8005

Talk is still on the schedule for Sunday 15h?

I keep getting:

The server was acting as a gateway or proxy and did not receive a timely response from the upstream server

in a popup nostr:nprofile1qyw8wumn8ghj7un9vfjkccnpwdjjuum0vd5kzmp0wfjkccteqqsqgc0uhmxycvm5gwvn944c7yfxnnxm0nyh8tt62zhrvtd3xkj8fhggyf5w3

Hi nostr:nprofile1qyw8wumn8ghj7un9vfjkccnpwdjjuum0vd5kzmp0wfjkccteqqsqgc0uhmxycvm5gwvn944c7yfxnnxm0nyh8tt62zhrvtd3xkj8fhggyf5w3 , My ditto keeps going unresponsive after it has been running a while. I see this when starting in the foreground:

ditto:relay PostgresError: duplicate key value violates unique constraint "nostr_events_new_pkey"

Could that be the reason? (latest in git today)

Replying to Avatar Alex Gleason

I want to tell you why #Ditto was slow and how we fixed it.

There were 2 issues: Postgres performance and nip05 lookups.

Postgres performance was caused by profile lookups, eg:

{ kinds: [0], authors: [a, b, c, d, ...] }

The issue was ORDER BY clauses on those queries. This caused Postgres to do a "sequence scan" instead of using the correct index:

"nostr_events_replaceable_idx" UNIQUE, btree (kind, pubkey) WHERE kind >= 10000 AND kind < 20000 OR (kind = ANY (ARRAY[0, 3]))

I actually fixed this already last year, and then reverted it in the last few months due to a bad decision. 🤦 See: https://gitlab.com/soapbox-pub/nostrify/-/commit/6a51c3f8a73c8a9f852a46d57119afdd4bc25b53 and https://habla.news/u/alex@gleasonator.dev/nostr-filters-limit

Logging query errors exposed this. This is why logging matters, people! Now I am switching Ditto to use JSON logging and incorporating Loki + Promtail so can easily track these problems in the future.

As for nip05 lookups - Ditto actually does nip05 verification in realtime as feeds are being rendered. This is obviously very bad. It does have an in-memory LRU cache, but restarting Ditto resets it. As a result, feeds can take up to 3 seconds to load (that's the timeout I set for fetching nip05). And if fetching fails, usernames are not displayed correctly.

To solve this (with a bandaid solution), I configured squid (HTTP proxy) to aggressively cache nip05 responses with this:

refresh_pattern /\.well-known/nostr\.json 3600 0% 259200

I also had to configure Ditto and squid with self-signed SSL certs so it can mitm and actually cache:

http_proxy="http://squid.lan:3128"

https_proxy="http://squid.lan:3128"

DENO_CERT="/etc/ssl/certs/squid.crt"

My TODO is to cache nip05 verifications in Postgres, to solve the root problem for self-hosters and increase performance even more.

tl;dr Ditto is fast now.

Only on your instances? Or will self hosters like me get the improvement s

Replying to Avatar jack

Morons. I hope you get to talk anyway.

Replying to Avatar TheGrinder

TikTok - from "spyware" to "national security threat".

https://www.yahoo.com/tech/tiktok-grew-fun-app-teens-130500533.html

I feel like any website, app or even game coming out of China is now being instantly labelled as spyware or worse. Meanwhile the NSA is harvesting more data than ever and the exiting US administration was more of a threat to the national security of the US than anyone else.

You cannot make this shit up.

Exactly. Your phone OS is spyware you can't remove or contain (unless you replace with a custom ROM). But TikTok is a problem? What a sick joke

I remain convinced it contains a backdoor. We already found out group messages get transmitted decrypted on reports. I am sure they can trigger the same for one on one

Jobs that would have been private anyway. Also they enjoy endless wars, surveillance and indoctrination in government camps called schools.

It's like being able to drink to much and tomorrow someone else has the hangover. All benefit and externalized consequences...