I made a dumb. I was in the duress pin setting, and I thought it was asking me for the current duress PIN. You can guess what happened next.

Recovery from backup is super mixed. Firstly, the thing cannot properly scan the taproot and segwit address ranges properly, so it misses UTXO's. I had to import the xprv into sparrow into two separate wallets (segwit and taproot), and then manually sweep the UTXO's that it missed.

The remote encrypted SCB actually works, and you can even zero in on which of the recent backup you want.