the secure component on the ledger is a seperate chip which runs a proprietary software.
for various reasons it is not possible to run opensource software on a secure chip.
there are arguments from highly specialized security software engineers that a security chip running a proprietary security software is safer than a hardware device running opensouce code because opensource code can be analyzed and exploited by attackers.
at the end of the day it comes down to trust.