nostr:npub1pvz25xfsvkm4rha9efxmjxmcfj5l40exj3vxf9ku4p5uskp6z8qqlzvndp Well with robots.txt you can `Disallow: *` and then allow specific crawlers or you allow everything and then disallow specific crawlers.

If you want to be found by search engines, most of them offer a manual way to be added to their index by uploading a sitemap or just giving them a URL.