Interesting. Lists could be optimised in the backend by indexing the pks and storing only the lists of indexes.
Discussion
I do this for tags. The issue is relays serve json, and unless you store the json in a ready format, generating json events on demand is very computational from lots of refs/joins.