Yes, of course you are right.

It doesn't work without trust. The Alby account is protected by an unlock password and an user password and can then manage the nsec. So, as you rightly said, it is better to set the nsec once in an account than to always use it pure everywhere. Of course it's not 100% sure.