I was a vegetarian for a really long time based on this thinking.

The ethical basis for these kinds of altered consumption patterns are extremely hollow for the reason you point out - there is no such thing as a diet that has zero impact on other creatures. We can minimize that impact but it will always be there. If you are alive, it's because other beings died for your benefit.