The codebase can be smaller, but I don't think it will be small enough to please your paranoia.
After all, you would still need screens to display the wallet and allow the user to set it up and manage it.
If you want to support multiple accounts, that gets worse.
Amber's codebase is mostly big because of the NIP-46 bunker implementation/notifications, which I think can be moved to a different app just for that with the use of Pokey to receive those events.
That being said, Amber is not supposed to be the only signer available. Other signer apps should exist and compete with Amber for a place in the user's hearts.