Yes, you can use IP directly. I am not sure about the CA cert, but you can definitely do CA on the IP level and not in the domain level.
But my main point is that clients need to find events. And the way we do it today is based on fixed relay sets. Nip65 allows people to migrate from relay to relay AND tell everyone following them where all the events can be found, similar to an update in the DNS registry to a new IP.