What hardware wallet and app interface are you using?

You should be able to load up the xpub address in an app like sparrow (for Mac/pc) or nunchuk for mobile, and you can derive addresses that way without having to reconnect everytime.