The initial difficulty adjustment would have been just an estimate, so initial block times would have varied a lot as the network hashrate varied a lot on a relative basis initially.

Once hashrate grew, the difficulty adjustment becomes more precise.

It's really no different today. Block times would have been way off for instance when all the China miners went offline until the next adjustment.