That's the way. There isn't any reason nostr couldn't be just, say, a thin UI layer in top of IPFS, if IPFS had a usable, scalable design to begin with.