It's all about who do you trust more. Sure fdroid is one more middleman, but they are a trusted agent in the FOSS community and provide valuable package curation and oversight.

There's no such thing as trustless binary distribution.