First, they need physical access

Second they need to know your pin of the yubikey

Still wouldn’t put my private key on there but no need to panic.

Just remove the keys now and use a proper HWW