Looking at my code it wasn't handling HTTP status codes from the relay at all. So I made changes. Now, on many serious errors including unauthorized, forbidden, not found, payment required, etc, gossip doesn't retry.