I thought about this pretty extensively I think it has more to do with that when we are first born we have only comparatively a short life span to compare the current time that we are experiencing. So it can seem that the time we are experiencing is long and drawn out which is why we always have a nostalgic feeling for our younger years.

So as we age we have a larger time frame in which to comparatively feel like we have actually experienced so as we get further down life's path of age that time seems to begin to accelerate because we have more experience under our belts of time felt. So this can give the effect of the acceleration of the same time being experienced.