You can make this deterministic and completely client side:
1. Query bootstrap relays for kind 10002 and 3 events. Extract a list of 100s of relays from these. Takes a second or two when I tested it.
2. Hash every relay URL.
3. Hash the user's npub.
4. Use Kademlia XOR-distance to find the 5 relays "closest" to the npub.
5. Use those relays as inboxes for the npub.
6. (Can update this periodically as the global relay set changes.)
An advantage of the deterministic relay set is you can find the inbox relays with just the npub. Solves one of the issues you appear to have with geo-relays.
PoC: https://github.com/chr15m/nostr-dht
Enjoy! 🧐
nostr:npub1sg6plzptd64u62a878hep2kev88swjh3tw00gjsfl8f237lmu63q0uf63m