If it’s really bad, (like one dice roll) you can expect your funds would have been swept within a week maximum. However, there can be less ideal situations which may still warrant a switch.
I used to be a Ledger Live user, and even though I had addresses which hadn’t moved in years, I still switched seeds and swapped addresses when I moved to a new security device. I could not verify how Ledger generated entropy, so I decided the risk was big enough that it was worth being extra paranoid and taking action.
In most cases though, if you’ve had a cold storage address which you haven’t interacted with in months, you’re probably fine.