How can you build a public Indexer for silent payments? unless you mean self hosted Indexers?
Discussion
The BIP352 has a section for Light Client Support.
Basically what a light client needs to detect an incomming Silent Payment is to calculate the sum of the inputs' pubkeys, that's all. So, if all the spending inputs were segwit or taproot client would be able to detect payments independently without the help of any indexer. However, for legacy scripts like P2PKH extracting the pubkey requires access to the prevout's scriptPubKey, something that unfortunately light clients don't have. That's basically why we need an indexer that provides the tweakData (in short, the sum of the pubkeys)