I think a lot depends on where you think the art came from in the first place. If you believe the art emerged from some sort of Platonic world of forms then the artist could be seen as more of a courier than a creator and it'd be much easier to separate the two.

Take comedians for example. Jokes, though considered art, are highly mathematical in nature, and if you're okay with continuing to use the mathematical discoveries of a mathematician who turned out to be an utter dingbat—because you feel the math in question in a way existed before that person anyway—then maybe you should also be okay with continuing to laugh at the jokes of say Bill Cosby.