All office phones should have a button that allows you to turn off the voicemail light without actually logging in and deleting the voicemails.

This post brought to you by the Post-It that lives over my voicemail button.