Passports standardized travel papers. They didn't invent them.

You used to have letters with a seal, from a local authority, granting or requesting safe passage. Most people never needed one because most people never traveled. People who wandered about, regularly, had special uniforms to clarify their status. A lot was done with clothing, as it was prohibitively expensive, so it wasn't prone to imitation.

If you traveled without papers, you would have to avoid gated towns, as they would question you at the gate and sometimes incarcerate you. This was very lucrative because you could be ransomed, such as happened to the Lionheart. Without a valid passport (or visa), you have no legal right to entry and could be detained and sent home, such as at Ellis Island.

Passport, the word, originally referred to landing papers at harbors, that allowed you to dock, trade, or even to travel freely inland after disembarking. You were, in other words, allowed to "pass the port".

What an EU passport does is allow you to travel across almost all countries on the planet, without having to ask at each one for safe passage, as the negotiation was done at state level.