to build search engine that actually finds content, it needs insider in every closed platform to perform search. and then that platform would probably notice this crawler, rate limit it and so basically make its task of indexing that site impossible