a flurry of discussions and things relating to my work lately have led me to something
there is two current existing query types in nostr, the REQ and the COUNT
REQ has no concrete notion of signalling how many events there are in a request beyond what it has been hard coded to limit results to
COUNT doesn't have metadata to easily determine other than making multiple COUNT queries
"since" and "until" fields in filters can be used to create a boundary that limits the number of results from a REQ but it is inadequate
i could swear i already made this suggestion before
but i'm gonna make it again, if i did, or for the first time if not
there should be a query that just spits back a list of all the event IDs that match a query, and if you set no limit, it just returns the whole set
if you consider that some follow events take as much as 512kb of data, and this is often a common size limit for individual events, then this is good for somewhere around 14000 individual event IDs to be returned in a result, it could be as simple as an array, so overhead is ["
perhaps this is not sufficient though, maybe you want to include the timestamp next to each event ID... or maybe you could make it so you define the first timestamp of the first event ID, and then after that it's seconds offsets from the previous, this would mean that the list would be something like
[[12345678,"
i'm inclined to say that fuck the datestamps, i'm just gonna make a new req variant that returns the IDs instead of the results as an array, and to keep with the style, it will just be
["REQID","subscriptionID","
the relay already can specify some size limit according to the nip-11 relay information, so it can just stop at just before that size limit and the user can query for that event and get a timestamp "since" to use to get the rest
nostr:npub1ntlexmuxr9q3cp5ju9xh4t6fu3w0myyw32w84lfuw2nyhgxu407qf0m38t what do you think about this idea?
if the query has sufficient reasonable bounds, like, it is very unlikely you want more than 14000 events over a period of let's say, the last day, of a specific kind, and certainly not if you limit it to some set of npubs
but you would still know where the results end, and so long as you stick to the invariant of "this is what i have on hand right now" the question of propagating queries can be capped by the answer of "what i have" and it is implementation internal whether or not you have a second layer and if you then go and cache the results of that query so next time you can send a more complete list
and i am not even considering this option
what about if instead of returning the results encoded in hex (50% versus binary hash size) but instead send them as base64 encoded versions of the event IDs, that gives you 75% or in other words expands the hypothetical max results of just IDs from 14000 to 21000
