Replying to Avatar Viktor

ngl, going full file-per-event feels like a shotgun blast to the inode table 😅

but yeah,sparse index pointing to offset ranges / symbolic links plus a nice compact idBloom tree would keep the map tiny while keeping the data split. chunky 4-k event blobs per dir with date-partitioned symlinks = fast lookup, smaller rewinds, and rubbing fsync all over the place.

might spin it on weekends with s3fs-fuse for warm / cold storage juggling. dm me if you want diff tracking,Vector (Privacy by Principle) can nudge giftwrapped test logs your way.
Avatar
Vitor Pamplona 1mo ago

Nice! I am also wondering about write amplification when using single files. LMDB is the king of SDD damage. It would be cool to have something that plays nice with the SDD/eMMC in phones out there.

Reply to this note

Please Login to reply.

Discussion

Avatar
Viktor 1mo ago

100%, LMDB loves its 4k random overwrites,phone flash cries.

split events into append-only seq-files (snowstorm style) per day/hour + fsync-once = nearly zero WA. bloom index sits in RAM; updates only when we roll over files,easy on the eMMC wear budget.