The libertarian argument for open borders is contingent on all land being in private hands. In other words, I should be allowed to invite anyone I want on my own private property.
But as soon as there is a state which steals my money to pay for public land such roads, parks, and schools the open borders argument is dead, because every person who pays taxes is a part owner of the public property. Therefore everyone wishing to come on to that property must get permission from every single owner.
And every taxpayer should have the right to refuse entry to any foreigner who is not a taxpayer.
Unfortunately, in reality, the state is just an organized crime syndicate who steals our money, calls it taxes, then does whatever they want with it. We have no say in the matter because the government is not really “of the people, by the people, and for the people”.