Learning more.

The breakout cable connection CAN work... it's just completely unpredictable and unreliable.

So new theory: the cables just make unreliable connections or are prone to interference. I moved the GND cable to a different GND and it seemed to work better, but could just be coincidence.

Now experimenting with slowing down the baud rate to see if that helps.