We don't need to. We should. All you're doing by keeping them is not getting a full picture of what's in the mempool and inaccurate fee estimation because of it.