Another obvious risk is to use your key on a non tested (or shitty update). The key can be leaked. A vault manager can reduce (or eliminate) this risk.