I can see why you are irritated, because your kids have less space to play. Not like they can go on the other structure. Whereas the dad could go play with his big kids on the big kid playground.

But I do think it is sweet he is actually playing with them. Maybe they'll migrate over if they relize there are more little kids around. I would just focus on making sure my kids get their turn on whatever it is they feel like playing on.