Oh for sure script validation is a more "real and present danger". I'm discussing more theoretically; the "state" is the utxo set, and theoretically you need all of it to do validation. Blocks, you don't. I think cardinality is relevant, though I'm not sure in detail, based on the size of a utxo serialization being roughly constant.
Lookups in a set aren't free, so a limit must exist somewhere, right?