You are correct. The key itself has a maximum security of 128-bits (although the key itself has a length of 256-bits)

This means that the passphrase is merely a 2nd factor to help with security in meat space but does not improve the underlying security.