The above idea works within a mint but it also works the same way across different mints.

Tokens don't have to be fungible, the users need to agree that the trade is actually worthwhile. The mints don't care about this happening (they can't stop it from happening, if they have HTLCs).