You need the &xs= parameter pointing to the URL of the torrent file otherwise it will only work if someone is seeding (because magnets themselves do not hold the torrent metadata, only a hash of the metadata, and the metadata is required to verify the downloaded pieces, so it must be fetched first from either peers or from the &xs= URL).