You never know when somebody might social engineer your password, SIM swap you, and take your Yubikey to a lab for half a day.