Read "Investigating the Federal Income Tax" by Joseph Banister

(Not legal advice)

I think you're something like 10x more likely to be audited if you file than if you do nothing

There are a few reasons they don't just send you a bill for what you owe in taxes like some other countries would - they want you to inform on yourself plus they are lazy

Income tax forms are "admissions of guilt" and you have a constitutional right to not incriminate yourself. You are technically guilty of perjury if you make any honest mistakes on your tax forms (even if you don't get prosecuted)

They make way more money per dollar they spend on workers investigating tax filers than they do on non-filers. The people trying to "do the right thing" are punished.

Tax code in the income tax section also has a weird (but intentional) definition of "the States and the District of Colombia" where "the States" only refers to DC (meaning it says it applies only to DC, twice). In other areas of the tax code such as that of petrol, "the states" is clearly defined as the 50 states + territories, showing that it's not some mistake.

There is Supreme Court precedent basically saying that you cannot assume legal definitions (like how one would normally assume that "the states" means the 50 states) and that everything must be explicitly defined. See Meese v. Keene, Colautti v. Franklin, and Western Union Telegraph Co. v. Lenroot

They barely send out any letters nowadays anyway

If they send a letter and it doesn't tell you an actual amount you owe, throw it away.

There are many levels of appeals as well

Tl;dr don't file and wait for them to tell you how much you owe then decide if you want to pay anything. Best would be to appeal until it becomes a great loss of resources for them to hire attorneys to go after you.

Yes but those letters could have a response that says “you have been charged 1million in fees and a 30% annual charge until you pay them back or face prison eventually..

This article shows us how many are not filing but not the repercussions.