Yeah it’s one of those frustrating cases where the obvious distinction is the right one, despite elaborate mental gymnastics to the contrary.
This “it’s all IOUs in the end” argument falls apart because it leads down a slippery slope the bottom of which is the recognition that all money derives its value from human imagination, and you can’t just stop halfway down that slippery slope and set up camp.