Completely reasonable question. Signet has something like 13 components on it. Yeah, that's it.
With one exception, they're all simple conponents (diodes, resistors, caps, a couple switches, a voltage regulator, and a USB connector).
The exception is the STM32, which is the only component to be concerned with (assuming there aren't any "extra" things on your board that shouldn't be there, which is easy fo verify with a visual inspection).
So this STM32 CPU, how can we trust it? To some degree, you're right, it could have some vulnerability in it from ST Microelectronics (the manufacturer of this particular model).
However, you can again inspect the board and see that when the slider switch is put in the "bootloader mode" position, it grounds out a particular pin on the CPU (after going through a resistor). If you look at the datasheet for the CPU, you'll see this puts the CPU into bootloader mode where it will not execute any code, but instead wait for a new firmware image to be flashed onto it.
If you compile and flash the firmware on there, that's about the highest level of guarantee that I know how to give you.
Yes, I've spent some time thinking about this problem. π
It also might be reasonable to compare the solution above with whatever solution you currently have for password management. Is the possibility of a backdoored CPU a higher risk than what you do now? If so, stick to whatever you've got! Also, please share whatever you have with me, because it's clearly awesome.