The modem shares the same infrastructure

The baseband nor any other radio does not have access to the RAM on the devices. They are isolated components via IOMMU and they're explicitly part of GrapheneOS' device's security and hardware requirements:

https://grapheneos.org/faq#future-devices

Cellular radios have always been isolated in GrapheneOS supported devices, even down to the first two devices we supported (Galaxy S4 and Nexus 5). It's a misconception cellular radios have privileged access to the OS or memory.

A long time ago Broadcom Wi-Fi radios worked the same way it did on laptops without having proper isolation but that was resolved on the Nexus 5X, and was never an issue on Pixels to begin with. Cellular was never more privileged than Wi-Fi, and in fact on several of the early devices Wi-Fi was not properly isolated via IOMMU (such as the Nexus 5, Nexus 9 and Nexus 6P), but cellular always had been.

Someone knows their stuff here.