Security relies on min-entropy, not Shannon entropy. Min-entropy bounds the worst case: the probability of the *most likely* interpretation, not the average.
Since H_min ≤ H_Shannon, proving high Shannon entropy guarantees even the adversary's best guess has astronomically low probability. We're not trusting averages, we're using the average as an upper bound on the maximum.