Inflation means a decrease in spending power. This doesn't happen as a result of new Bitcoin being mined.
New Bitcoin enters circulation, less and less each halving as you said, but the total amount of 21 million is priced into the value of BTC.
The fixed supply is why it isn't inflationary.