It *would* be different if someone could brute force your seed phrase. In that case, adding words would definitely improve security. Since brute forcing a seed phrase is currently intractable, we have to assume that the attacker stole it. Presumably they would also steal your passphrase, but if not, it needs to be a pretty long, non obvious passphrase, or it will be trivial to brute force.

Now – is it more likely that your seed will be stolen in a way that makes your passphrase a useful defense, or that you'll someday forget it and lose all your funds?