I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.

Potentially:

Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)

Lost all zap.stream balances (history / balance)

Lost LND wallet balance 10M+ sats (searching for wallet seed)

Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)

Lost all code hosted on git.v0l.io (No backups, gone)

Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..

At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...

The last good database backups i have are from Feb 2024...

Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..

Reply to this note

Please Login to reply.

Discussion

Damn man that sucks. Let me know if you need to any help with anything

๐Ÿ˜ฉ๐Ÿ˜ฉ so sorry! Thatโ€™s so stressful & frustrating

thoughts and prayers.

I'm guessing no streams in the foreseeable future?

Mistakes happen. ๐Ÿซ‚

Ah....I think it was a minimum of a million sats.... ๐Ÿ˜œ

Jokes aside sorry to hear about this! That's horrible ๐Ÿ˜ž

Sucks - but no risk no fun. Thank you for running all these services for the community.

love this attitude when you're holding other peoples sats

Oh damnโ€ฆ best of luck to you.

Sorry to hear mate! Donโ€˜t be too hard on yourself, mistakes happen. Iโ€˜m crossing my fingers for you ๐Ÿคž๐Ÿป๐Ÿ€๐Ÿซ‚

Maybe someone has remotes of the git repos?

I think he mirrored at least some repositories to GitHub, but presumably the tracked issues from his Gitea instance is gone.

Yeah, issues are tricky. Contain so much business knowledge.

We're trying to figure out how to mirror them over Ngit, through the API, so that we have them backed up on the relays.

Respect to you for the full, frank disclosure, Kieran. You've been carrying a heavy load. I appreciate you. ๐Ÿซ‚

Don't worry. You make it through this too. Plebs are with you

Let that be a lesson to you plebe devs

#boatingAccident nostr-style

I'm just joking.

Take a deep breath, don't panic, try to calm down. There's usually one or more way to fix things. You'll be ok.

Appreciate the transparency and honesty. Weโ€™re in this together on a new frontier with risks and all.

๐Ÿซ‚

Good luck Kieran. Youโ€™re solid for being straight up.

Tough but Iโ€™m sure there is a lesson in there somewhere. ๐Ÿซ‚

be good Johnny

Remember #reckless ?

Don't worry, breathe, it's not a question of life or death.

Thatโ€™s a hardcore loss.

Iโ€™m so sorry to hear about this fatal flaw ๐Ÿซ‚ data loss is no joke.

As you rebuild, donโ€™t treat yourself bad. This too shall pass.

Aah sorry to hear that Kieran - fingers crossed you recover as much as possible - either way I'm sure you will rebuild stronger.

RTFM - "Use extreme care when adjusting the volume size."

tl;dr - I had an 8TB disk in proxmox that only had 2TB of data, wanting to free space, i used `zfs set volsize=5T` on that volume, after repairing the filesystem, only 1TB of data remains (mostly bitcoin node blockchain data and nostr.download data)

I will continue to attempt data recovery to restore those critically missing databases.

On a side note, are there any data recovery experts on nostr? #asknostr

Hello, I used autopsy but I'm not an expert. Autopsy works for restoring all kind of files and its very easy to use. Even I can use it ๐Ÿ˜…

Not an expert, but I've had good results with photorec. It's made to get files off of severely damaged disks. IDK if it can recover DB files, but worth looking into.

I second this, photorec saved me a couple of times.

I have also saved my own ass with photorec.

I've never heard of Photorec.

I have a hard drive with a ton of files and photos on it that will not open.

I took it to a few different places but they were going to charge me an insane amount to recover (if even possible)

Can you share more about Photorec and any relevant advice?

Thank you ๐Ÿซก

1. Make a copy of your drive ASAP. If it's failing, you may be making things worse just by trying to use it. ddrescue is designed to do this for failing drives. This may be all you need to do, but probably not.

2. Read the documentation for photorec or any other tool before doing anything. It's easy to make things worse if you don't know what you're doing. https://www.cgsecurity.org/wiki/PhotoRec

I have never run that command and now never will. Thanks for the heads up

I think the bigger lesson here is: donโ€™t put ALL your systems on one hard drive, even if they are backed up.

Oh, and make a block level copy of the disk to work on so you don't make things worse.

It might be worth asking how much professional data recovery would cost vs. time and sats lost at this point. If you drives are still working, it should be relatively cheap.

Hey nostr:npub1v0lxxxxutpvrelsksy8cdhgfux9l6a42hsj2qzquu2zk7vc9qnkszrqj49 get in touch with @Albercoin in twitter (he does not have NOSTR). He offers self-host solutions un #proxmox and is a master.

nostr:npub1yn3hc8jmpj963h0zw49ullrrkkefn7qxf78mj29u7v2mn3yktuasx3mzt0 nostr:npub12aqfngts2xc0z0n47lyfx03p8prhz4kqcel29mukgwydeuatawqqg5dvgy igual podรฉis ayudar aquรญ.

Does it's HDD not SSD?

if you have technical details that you can share about this thing happened, it will be great if you do a write up about it.

Donโ€˜t beat yourself up โ€ฆ now you have the chance rebuild bigger and better.

I have most of my git.v0l.io projects mirrored on GitHub. Gotta check the remotes.

how does this affect already running lnvps instances?

No impact RN for lnvps users, just need to reach out to customers to re-create the database at some stage (which vms belong to which pubkeys)

VM's will be left running with no charge until we get back this mapping

๐Ÿซ‚ these things happen, brother -- don't beat yourself up

๐Ÿซ‚๐Ÿซ‚๐Ÿซ‚

iirc, I had a balance of 25M bitcoins in my zap.stream account, pls zap it back thaaanks

Oh well I only had 24 million bitcoin there... Can I get mine first? ๐Ÿ˜…

At least you didn't send our private keys to Bugsnag, right?

Just kidding, but I don't think this is super bad. The worst part is the money you lost on LND, you don't owe anyone anything.

The money isn't lost yet. (Announce you don't have a watchtower up for revocation secrets on the internet isn't a great call) If everyone just closes the channels with you, you get those sats back to your on-chain address. Minus fees, still seems okay.

agreed

You being mad at you is like wasting time. Treat yourself like you would treat a child.

After you get things fixed MSG me.

Maybe i can help with backup or Server or or or

And not to pile on, but i cant even zap you ๐Ÿ˜”

โค๏ธโ€๐Ÿฉน

Remaining data appears to be complete garbage... rip everything

Shit happens. Thanks for the transparency and honesty. ๐Ÿ™๐Ÿผ Hope you get up and running again soon!

you do a lot for nostriches. mistakes are bound to happen from time to time. best of luck on recovery, we got your back dude ๐Ÿซ‚

๐Ÿซ‚

Yeah I noticed karnage@zap.stream disappeared ๐Ÿคฃ would love to have that back.

Have you checked on mastodon.

Alan Jude or some of the other zfs wizards

Definitely a bad day, but no one was hurt, it's just tech stuff; so take the lesson and move forward with patience. I only hope you can recover your money. If you need help, ask, we are here as a community.

Canโ€™t seem to zap you. Invalid address

I tried tooโ€ฆ

we appreciate all you do for nostr ๐Ÿซ‚

๐Ÿซ‚๐Ÿงก

Sorry man. ๐Ÿ˜ฉ๐Ÿซ‚

nostr:npub1v0lxxxxutpvrelsksy8cdhgfux9l6a42hsj2qzquu2zk7vc9qnkszrqj49 , donโ€™t worry, we understand that such mistakes can happen. We are all human.

You got all our support. You will make it right, one way or another, i am sure. ๐Ÿ‘๐Ÿผ

Keep up the good work Kieran, you're doing great ๐Ÿซ‚๐Ÿ’œ

Youโ€™ve got the whole of Nostr on your side buddy.

๐Ÿซ‚ Don't worry, we're all with you. It happens.

I sincerely hope things work out alright for you.

Sorry to hear that man. let me know if there is anything I can do to help

On the bright side. nostr.download going down is forcing me to work on blossom / nsite stuff and make it more (self) censorship resistant

As a dude named Ben, I know the feeling of that moment you accidentally clicked the wrong thing and production goes down. It feels awful and I'm sure you're beating yourself up. It's impossible not to. I hope your recovery from this goes as well as it possibly can. The transparency is very much appreciated. Good luck!

wow rough

Hugs ๐Ÿซ‚

The good thing is that you will mean with your mistaken, awaes keep a offline backup and update daily or weekly

Iโ€™m so sorry Kieran! ๐Ÿ™

Let me know if there is anything I can do to help.

sorry to hear that. this things happen to big companies even. i appreciate your work as an individual doing all these stuff and keeping it up.

i hope you get the datas and moneys back soon. ๐Ÿซ‚

we are all one mistake away from doing the same

I have no clue what any of this means, but I once waded into the ocean to impress some girls to retrieve their frisby but did so with my phone in my pocket.

Did you get it and did you impress them?

I my mind I was like a taste wopper like you see of the menu, but in reality it was less than impressive

hope it was worth it... ๐Ÿ˜‚

in hindsight no. ๐Ÿซฃ

๐Ÿซ‚

damn man, sucks for you and obviously all the affected. Hang in there as you move towards making things right.๐Ÿ’œ

Shit happens.

When you say you lost the domains, do you mean you lost access to the registrar account?

Hang in there.

Damn, this sucks. Hope you can recover as much as possible and that it's not stressing you out that much.

๐Ÿซ‚

Damn this sucks but thanks for all youโ€™ve done for nostr

๐Ÿซ‚

Been there ๐Ÿซ‚

๐Ÿซ‚

๐Ÿซ‚๐Ÿซ‚๐Ÿซ‚

Not cloud, just run in others computer.

No problem, just a game.

๐Ÿ™

oh that sucks. lmk if you need a hand rebuilding things back

Was trying to start my first stream today ๐Ÿฅฒ we are here to support you!

I want to understand this ๐Ÿ˜”

Hang in there! Here to help however I can brother!

so sad. I love your site. It's not easy running big websites, with severs being so expensive having backups is difficult. It's happened to me before.

Don't give up.

My worst nightmare ๐Ÿคฏ๐Ÿคฏ, Iโ€™m so sorry, and if you need anything, Iโ€™m here for whatever you need.

STEADY LADS

๐Ÿซ‚

nostr:npub1v0lxxxxutpvrelsksy8cdhgfux9l6a42hsj2qzquu2zk7vc9qnkszrqj49 if you need any assistance with any of this dilemma, donโ€™t hesitate to reach out. I will reply to your DM. It would be a pleasure to assist you if required. You [and many others] work hard contributing to this ecosystem. Peace, love and hugs.

๐Ÿซ‚

nostr:npub1v0lxxxxutpvrelsksy8cdhgfux9l6a42hsj2qzquu2zk7vc9qnkszrqj49 Were dvr.zap.stream and data.zap.stream affected? I get 503 errors from dvr and 404 not founds from data.

Yea the dvr.zap.stream went down, all the data was lost for old recordings.

Aw man. Sorry to hear that