It’s important because (except for OTC) you can’t buy coins that aren’t on exchanges. Therefore supply is limited when compared to demand.
End result: higher fiat prices and more FOMO.
Discussion
I have nothing to support this argument but maybe:
The coins on exchanges can be seen as ‘for sale’ whereas the coins not on an exchange are ‘not for sale’.
You are correct that to transfer is trivial but the fact that they are not already on the exchange suggests that the holder is currently not interested in selling.
Maybe.