I distrust F-Droid, but I would never even think about going along with github
You are right.
99.999% people just trust the provided binaries.
To more or less reduce that worry, there is a neat feature on github that permits to compile the code directly from their docker instances and let you download a freshly compiled binary.
The geogram code for ESP32 is not so big, it is relatively easy to read for manual inspection: https://github.com/geograms/geogram-tdongle/tree/main/src
Discussion
The other option is compiling yourself. For a small project there aren't many other ways of making it more comfortable for users to perform these actions.