Do you see any speed drop due to this?

I guess you could cache the npub for 30 minutes or so, so you don't need to do the validation all the time. Just check the cache and invalidate every 30 mins.