Mark them as inactive if their last event was at some arbitrary past date, then only check inactive pubkeys every once in a while (random). If a new event appears, mark them as active again.