I never really noticed a big impact from federation; but I did notice a very, very noticeable impact when joining huge rooms with many federated users; i.e. sone of the bigger ones at matrix.org. That can, and will, become more of a benchmark than a usable situation really fast XD