3gb in the working set or just virtual memory (committed)? I don't understand enough about the Go GC but I wonder if Valgrind might be able to help in a controlled test executable?
Discussion
i'm tracking the location of allocations using pprof... i have to let it run for a while, and then i can see in the system monitor that the memory utilisation keeps on growing, pretty much linearly with the amount of queries coming in, and indeed, the memory allocations are being reported in a certain place and they seem to not be getting deallocated
it's probably some dumb shit on my part but this has been quite a wild goose chase for the last week lol
Ah, so is this similar to dotnet where a reference is hanging somewhere and the GC can free? There is also a nice dotnet tool for tracking that too :)
yeah, i guess C# uses GC as well
this case is really problematic for me, i mean, it just blows out memory
on a smaller system it led to heavy CPU use managing the GC, now it just blows out the memory without cpu use, until it hits the ceiling and then it OOM kills
i've been fighting with this one some time
i have now even found there is an update to badger but i'm kinda losing faith in badger at this point
the profiling is clearly showing that the memory is not being disposed of properly by the database
oh well, i will keep on chipping away at this, i'm sure it will be resolved soon... i'm getting really familiar with the areas of the code where the problems are now