A malicious miner can come in a few forms:

1. They are trying to censor the network and have sneakily amassed over 51% of the hash power so it is working.

2. They are attempting to create blocks with invalid transactions or just plainly not following rules yet attempting to pass off their data as a bitcoin block.

Thats probably most of the cases, othet cases kind of fit into one or the other.

In case 1, we "kick them out" by introducing a new rule and roll it out. This isn't a simple task, it will involve convincing people that this is the most conservative way to work around these people while not leading down a slippery slope of just introducing a way to censor people.

The other way we kick them out is by rallying the people to run more nodes and mjners, to outmine the bad actors.

Eventually, the cost to censor will be too much and the enemy will run out of funds.

For case 2, the system already works. Even if the malicious miner had 90% hash rate, our software rejects anything that is considered invalid so these blocks are ignored by users. Some users who rely on custodians or third party nodes may be affected, but majority of economic activity and therefore the most fees to collect will only be collectable by miners who also don't mjne on invalid blocks.

The attacker will quickly realise they are wasting time, energy and money and even if not, we are never affected and more people who use custodial systems start to learn to use Bitcoin natively.