Yeah, when you're only doing it for a single user it's really not too heavy, you end up with maybe 10-20k pubkeys in the graph, everyone else has a score of zero. The private mutes thing is a limitation, no way around that except something much more complex like ZKPs or some kind of request/response protocol for requesting that information between pubkeys.