Generally I don't think one should leave cold storage secret key material on a signing device.

Good question.

Redundant storage is orthogonal; the signing device is a bad place to store the secrets.