Yeah I'd probably look for a softer one. It just depends on how long you want your filters to last. I'm just taking sediment out, so I want them to last for as long as possible.
We have a dedicated scourer that doesn't get used for dishes (ideally you don't want detergents in contact with your filter candles).