Experimenting with bit magnet as a personal torrent indexer.
So far, I've used DHT to crawl over 150,000 torrents and have them all indexed locally, which is kind of cool. And it's not actually downloading the torrents themselves. It's just indexing their metadata.
According to bit magnet itself, it takes roughly 80 gigs to store over 150 million torrents.
What does it do exactly? ELI5 please
^ Im curious too. You were helping me understand torrent stuff a little while ago but I'm still a little too busy for it :)
So essentially it's a private BitTorrent search engine. It uses DHT to discover torrents, its saves their metadata and then tries to classify what they are, such as like music, programs, videos, things like that.
The only downside to it, though, is because the app is relatively in alpha. There's not a lot of filtering you can do on what kind of torrents you collect. It literally takes in everything. So, like, out of the 150,000 torrents I have indexed right now, probably about 20,000 of them are porn.
According to the docs, there are some ways to tune it, and I haven't really looked too closely at that yet.
So you are basically crawling magnets and storing them so you can "search" them on demand? Can't magnets change to update trackers and stuff if I understand that correctly? Or do they remain rather constant?
It periodically re checks the torrents you have indexed. It also has some basic availability tracking Like how many people are seeding and leaching. I think there's also a threshold you can set for how long you keep stale data before giving up on it.
Hmmm. I wonder how much network noise that makes? Like how many devices will see my IP address or is that all protected by tor?
I have a VPN on the edge of my network, so I'm not really too worried about that. But it's definitely not protected by Tor.
Same but I use the same VPN for many things and background services all use the same VPN so you can hide your physical location but not your activity.
I guess technically there's nothing illegal about doing indexing, but some overzealous and rudimentary anti-piracy tools might flag you. /shrug
It's been running for less than 24 hours. So we'll see if anything bad comes from it. And I'm probably going to take some time tonight to tune my classifier and fiddle with settings so I can try indexing less porn.
I'm at about 210,000 indexed torrents and 24,000 of the classified torrents I have are porn. I would like to cut that number down.
True but I like to live by "Give me six lines written by the most honest of men, and I will find something in them to hang him." If they can't find six lines I'm safer.
I'd be more worried about the copyright material than the porn (CP aside of course) if you are in the US XD. They like to kick doors in for that too now days.
Yeah, I'm definitely more concerned about copyright stuff rather than porn besides the obvious problems there. 😅. We'll see if I get any DMCA pings here in the next few days. I would assume they would have already pinged me, because they seem to be pretty damn fast about that stuff. (i.e. somebody came over to my house, connected to my guest network, and was seeding a copy of Shrek a few years ago, and that was the only DMCA ping I've gotten in close to a decade. And it happened within a couple hours of them connecting to my Wi-Fi.)
Wow no kidding. I only ever got a notice quite a few years ago. Forgot the VPN to my mom's place was connected... She sent me an email about a week later XD
Fine, I'll build my own torrent site 😙👌🚀
It's kind of cool so far. I'm hoping it can help me find torrents of things that have been hard for me to find on both public and private sites.
A bit of an update. Tuned the classifier pretty well. But it was taking forever to re-classify ~300k torrents, so I nuked the DB and started from scratch. it's nuking porn, and it's tagging anime correctly, and its finding more audiobooks (I "vibe coded" some updates to the classifier to make it a bit better.)
NGL this is neat as hell. Now I just let it run so it builds the index.
