People say that like it isn't an absurd statement. "Price" is merely a measure of what you can buy with your Bitcoin. Bitcoin would be pointless if it couldnt consistently buy more over time (a higher price). It has to at a minimum keep up with what fiat can buy. But realistically, most people wouldn't bother adopting it for an equal trade.

Price absolutely matters.

Even if fiat died, the "price" of Bitcoin would be set by a market of goods and would absolutely matter. No one would give a fuck how much Bitcoin you have in the absence of a price. Price is value. And it has to keep going up in fiat terms to at least match inflation. Otherwise, it fails. To say that doesn't matter is just retarded.