Subnostr

what a wild goose chase this memory leak bug i have in front of me has become

like a game of whack a mole... i find one hole, close it, and then i find another hole, close that, opa, another hole

now, the issue is appearing inside the badger code... almost 3gb builds up after about 10 minutes just doing regular stuff

it kinda looks like i'm gonna have to rewrite this query events code altogether, at this point, i think something about the structure of it is keeping memory hot for a long time that should not be kept

oh well, soon, i think #thoon™️

Reply to this note

Please Login to reply.

Discussion

Comte de Sats Germain 1y ago

3gb sounds like a lot. Is it possible to tell it to delete it all periodically while not stopping the process?

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

yeah, that's the thing about garbage collection, and the big weakness in Go's garbage collection has to do with what is called "escape analysis" - figuring out whether a value allocated in one scope goes upwards to the caller and so on

i'm hot on the trail though, i think maybe this evening i will finally have this bug squashed

so far it looks like part of the problem is i have a set of decisions that evaluate whether to ignore a given query result but that decision is not made inside the scope where it is created, and so the GC is holding the memory and not letting it go

i may have just fully nailed the problem now though, fingers crossed

ChipTuner 1y ago

3gb in the working set or just virtual memory (committed)? I don't understand enough about the Go GC but I wonder if Valgrind might be able to help in a controlled test executable?

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i'm tracking the location of allocations using pprof... i have to let it run for a while, and then i can see in the system monitor that the memory utilisation keeps on growing, pretty much linearly with the amount of queries coming in, and indeed, the memory allocations are being reported in a certain place and they seem to not be getting deallocated

it's probably some dumb shit on my part but this has been quite a wild goose chase for the last week lol

ChipTuner 1y ago

Ah, so is this similar to dotnet where a reference is hanging somewhere and the GC can free? There is also a nice dotnet tool for tracking that too :)

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

yeah, i guess C# uses GC as well

this case is really problematic for me, i mean, it just blows out memory

on a smaller system it led to heavy CPU use managing the GC, now it just blows out the memory without cpu use, until it hits the ceiling and then it OOM kills

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i've been fighting with this one some time

i have now even found there is an update to badger but i'm kinda losing faith in badger at this point

the profiling is clearly showing that the memory is not being disposed of properly by the database

oh well, i will keep on chipping away at this, i'm sure it will be resolved soon... i'm getting really familiar with the areas of the code where the problems are now