When I use a trimmer I trim weekly, it's fine. If I go actual razor bald I can go 4-5 days between shaves. Your scalp hair is much finer than your beard hair, so it's a lot easier to razor off. I spend less time on 1 shave as I would dealing with long hair every morning.

Your hair's growth rate isn't affected by it's length, it's constant. It just seems that way because it's easily noticable when it's extremely short.