Although you're correct, they will still attempt a Notice of Deficiency and if you don't know how to rebut that, you're fucked.

The key is finding an employer that you can show that filling out a W4 is not necessary and the IRS has provisions to protect employers from their employee not wanting to fill out a W4, because then the 1040 will not be expected (even though a 1040 isn't even meant for 99% of people by the IRS's own rules).