You can use your seed with every wallet. The cold wallet you have stores the private key in an encrypted form while the seed lets you generate this private key.

If you want top notch security, this would be pointless though because the main concern with hardware wallets like ledger etc. is, that they generate the seed for you, using an algorithm. This leads to the problems that you 1) have to trust the company producing these wallets that they don’t scam you and 2) that the algorithm won’t be discovered by a third party, because it would make it easier for them to guess valid seeds.

Basically what you said works, but it won’t increase security