There is such a thing as the sunk cost fallacy. But quitting because of someone else, is not good. If you don't come to the same obvious conclusion, then don't. Problem is: you need to respect some other person's opinion if you want to share in the outside perspective, and finding the right input is tricky. (At least, that's my impression.)

Also, "quitting" doesn't mean your losing your skill and experiences, so you never end up at 0.