Maybe its just a node on the path of your payments that is slow, not even necessarily your node that is slow. In theory nodes you route through or the receiver can hold the payment (htlcs) for hours.