you do not need to trust

do not upgrade it. you can even restore it without a ledger device.

the secure element is more secure than any other device statistically/mathematically