You forgot to mention that this is only possible when ledger sync is switched on. Otherwise your ledger is fine and no one can access your seeds.