I always tell my kids, "you can learn something from everyone, even if its simply to learn to know what NOT to do."

But it's true, just get those people out of your way. Don't feel bad. Don't look back. If they REALLY want to be, they'll be inspired by your success, but if not they will criticize it and try to crush it from their insignificant position forever.