It depends. What size are your withdrawals usually?
Depending on your wallet software you should be able to see how many UTXOs - or “Bitcoin chunks” you have.
If you have been withdrawing small amounts of Bitcoin frequently, you’re probably going to want to combine them into bigger chunks. If you have been withdrawing large amounts, you are in the clear.
This likely isn’t a crippling issue either way - but fee efficiency is important, especially if we assume fees will be 10x higher in the future. Best to adopt efficient practices now.