Hi all. Our software has been using jemalloc 4.x for 10 years or so and we've been sitting with the very latest commit on the stable-4 branch for the last 4 years. I've been idly toying with the idea of updating to the latest 5.x version. We made some small tweaks to 4.x but the one major feature we added was the ability to dump profiling to a memory buffer as an alternative to writing to a file: our software is a distributed system and we need to be able to collect profiling over the network in response to requests for metrics on different nodes in the system: writing to a file is not helpful in our environment. Of course, I can forward-port the changes but I was wondering if anything like this already exists in jemalloc 5.x, or if there are any thoughts on what would be a good way forward?
@madscientist You can use mallctl(“prof.dump”…) with a filename pointing to shared memory (e.g. /dev/shm).
Another direction, is that we plan to add a prof dump hook, i.e. a user defined callback invoked upon prof dump, which can be used to redirect the output. However it’s still an experimental feature (and will be unofficial / undocumented in the upcoming 5.3 release).
@madscientist In that case, you can take a look at jemalloc/jemalloc#2119 which added the experiemental prof hooks. Note that there are 2 hooks — one for backtrace (when sampling an allocation), and the other is the dump_hook which is triggered when output profiles.
Feel free to let us know if it fits your use case.
Thanks for the pointer. It doesn't quite work for me; it appears that this will still create a file and dump all the profiling to it, then it will invoke the hook you register with the filename. To use this, I would need to implement a hook which would open and read that file into a buffer then send the content over the network, then delete the file. Of course, this would work in general. However what I really want is a way to replace the entire write of the data so that instead of writing to a file it will write directly to a memory buffer, then when the dump is complete I will send that memory buffer over the network. I don't want the data to be written to a file then read back in.
@interwq I am intrigued by that PR for a different reason. The ability to insert a stack frame (as illustrated in
test_prof_backtrace_hook_augment) would be incredibly useful (e.g. to set some kind of context thread-locally from java), but I can't see how to make what you add show up meaningfully in the jeprof shell or visualizations.
@pnf the use case we have right now is integrating python frames with native frames — it does require quite a bit of work on the consumer side, e.g. building a python frame lookup database on the fly, then combine the results when post-process. But like you said it’s not part of jeprof. The end results are quite nice though: it gives us an unified view with both python and native stacks in the same dataset.
hi, I am new to jemalloc and trying to configure the build for my use case. When I set tcache_max in the --with-malloc-conf, it doesn't seem to take in to affect. When I read tcache_max via mallctl it returns 0 for opt.tcache_max and 32K for arenas.tcache_max. What is the right way to set this? Also what is the different between opt.tcache_max vs arenas.tcache_max
tcache_max is added in 5.3, previously it was lg_tcache_max and cannot be smaller than 16K (assuming 4k page size)
BTW I do have an higher level question. I am trying to compare tcmalloc with jemalloc. The memory footprint when using jemalloc more than. doubles compared to tcmalloc. What attributes are controlling this?
If you can obtain malloc_stats, I’ll be happy to take a look (e.g. from malloc_stats_print(), or use the interval based stats output in 5.3)
The options in https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md should be a good starting point. Depending on which metrc (memory vs CPU) you value more, there are several options. Usually decay and narenas are most effective. Also enable bg thds will solve some edge cases.
is there a way to compile in opt mode?
With narenas=48 , the allocated reduced to 7.3G with 3.0G resident
Oh please don’t pay attention to the VM size — since jemalloc 5.0, we started caching VM and also being generous on VM reservation. See https://jemalloc.net/jemalloc.3.html#opt.retain