Correct.

If your device fails (e.g. lost, broken), your only option is to get another one - even if you just want to immediately exit after you get it.

The app on your phone has the key to setup a new hardware key for your new device; and then cosign with the key on the server to move your funds from the old 2-of-3 keyset to the new 2-of-3 keyset.

The only purpose of the recovery contact is to help recover the 1 key that's in the app (i.e. in case you lose your phone)... you would still need access to at least 1 of the other 2 keys.