Well, it's true that you can extract the key. But if you wipe it after creating it, I think you have solved the problem. Just don't forget to erase it.
I don't have a good answer to the issue that I have to trust the secure element. That is why Coldcard uses two, so if one is compromised and the other is not, you are still good. But you still have to trust that they are not both compromised.