Correct. It's all just inputs and outputs. If you think you got dusted, move that input to a new address and be done with it. Probably cost you more than the dust is worth but it's worth it. Also, with some wallets you can select the inputs you put into a transaction. You could lilterally leave the dust in the address if you know what you're doing. Just make sure it never gets added to an output.