jemalloc/jemalloc

jemalloc development discussion

Brennan Vincent

@umanwizard

@interwq Thanks! Is the time of the official release still unknown?

Qi Wang

@interwq

@umanwizard Sorry it’s been on our plan for a long time. I’ll try to get to it later this half. Best guess is by EOY.

Brennan Vincent

@umanwizard

OK, thanks for the update.

mayadav1

@mayadav1

Hi, Can anybody tell that jemalloc supports windows?

Dave Rigby

@daverigby

@mayadav1 Yes, it does.

agupta-zz

@agupta-zz

Hi, mallctl call is failing with errno set to 2 (ENOENT). Could you please help with resolving the issue

malloc.conf -> prof_active:false,prof_leak:true,lg_prof_sample:17,background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0

David Goldblatt

@davidtgoldblatt

Probably you didn't pass --enable-prof to configure

agupta-zz

@agupta-zz

i just used the default configuration. Any other configure option i should use ?

David Goldblatt

@davidtgoldblatt

Nope; I think just prof is necessary for those mallctls to be available

agupta-zz

@agupta-zz

alright. let me compile again and try. Thanks for your prompt response.

David Goldblatt

@davidtgoldblatt

no problem!

agupta-zz

@agupta-zz

@davidtgoldblatt i compiled with --enabled-prof. Still get the errno 2 for mallctl

Am i missing anything ?

Qi Wang

@interwq

@agupta-zz you will need to enable the prof at runtime too, e.g. by adding env var MALLOC_CONF=prof:true

Vladimir

@0xdeafbeef

Can anybody help?

This is with MALLOC_CONF="narenas:2,abort_conf:true,dirty_decay_ms:1000,muzzy_decay_ms:1000"
INFO ton_indexer_test = RSS: 325.3 MB, Allocated: 305.6 MB, Metadata: 11.4 MB 
INFO ton_indexer_test = Mapped: 334.0 MB, Active: 315.2 MB, Retained: 418.3 MB 

This is with no configs at all.
INFO ton_indexer_test_mi = RSS: 1.1 GB, Allocated: 456.3 MB, Metadata: 27.5 MB 
INFO ton_indexer_test_mi = Mapped: 1.2 GB, Active: 496.9 MB, Retained: 2.3 GB

It's rust application using rocks db. Both are using jemalloc.
sizes from heaptrack

Btw, mimalloc is using 2Gb and keep growing.

Dave Rigby

@daverigby

What’s your question? why is the RSS and mapped bigger in second instance?

Vladimir

@0xdeafbeef

Yes. And retained

Dave Rigby

@daverigby

So for a start the default number of arens is something like 2x (4x?) number of logical cores, so assuming your machine has more than 1 core, you’ll have loads more arenas

(and so if you’re allocating a range of sizes across many threads your heap could get pretty fragmented)

I don’t know off the top of my head what the decay defaults are, but again I would suspect they are less aggressive than what you’re setting with that explciit config

Vladimir

@0xdeafbeef

Basic decay is 10s, I have 12 cores

Dave Rigby

@daverigby

ok, so you’ve got 48 areans there vs 2

If you set opt.stats_print to true, you can see the state of the individual arenas at exit, that might help you understand the differences

I suspect the difference in arena count will be the main reason - broadly it’s a performance vs space trade-off

(space / fragmentation)

Vladimir

@0xdeafbeef

What is the basic reason of fragmentation? Small values? Maybe there is a guide/doc for tuning it?
I've just set

narenas:2,abort_conf:true,dirty_decay_ms:10000,muzzy_decay_ms:10000

and got

INFO ton_indexer_test_mi = RSS: 1022.3 MB, Allocated: 501.1 MB, Metadata: 24.7 MB 
INFO ton_indexer_test_mi = Mapped: 1.1 GB, Active: 526.2 MB, Retained: 974.3 MB

So it's something with decay times

Both RSS and retained are growing

Dave Rigby

@daverigby

What is the basic reason of fragmentation? Small values?

You want to allocate someting less than a page - e.g. 16B. jemalloc allocates memory using pre-defined sizes classes - i.e. all 16B allocations will come from a set of (4kB) pages; 32B allocations will come from a different set. That makes the maths to allocate / free allocations cheap and quick.

But, that means that fragmentaion can be costly - e.g. first 16B allocation always costs 4KB, as you have to allocate first page

Similarly, if you allocate enough 16B allocations to require 2 pages, but then free half of them at random, statisitcally you’ll still have half the allocaitons in each page, so your RSS is still 8KB

multiply all that by the number of arenas, as each arena is independent...

Re: the decay stuff, I’m not super familar on the details but that controls how often / aggressively jemalloc looks at the heap and tries to unmap (from the process address space) pages which are currently unused etc

Dave Rigby

@daverigby

Dunno if there’s any newer writups, and certainly details have changed a bit, but https://engineering.fb.com/2011/01/03/core-data/scalable-memory-allocation-using-jemalloc/ is a reasonable overview

Vladimir

@0xdeafbeef

Thank you!

David Goldblatt

@davidtgoldblatt

https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md is the other thing we like to point people towards

vinayhudli

@vinayhudli

Hi All, I stumbled upon jemalloc while trying to debug my java app for native memory leaks. I'm facing an issue when configuring jemalloc specifically 5.2.1 tag, when I issue : ./autogen.sh --enable-perf
I get configure: WARNING: unrecognized options: --enable-perf .
Appreciate any help

David Goldblatt

@davidtgoldblatt

Hi; it's prof, not perf

vinayhudli

@vinayhudli

yeah I just now figured that and realizing now that I should not have posted this. Thanks @davidtgoldblatt

David Goldblatt

@davidtgoldblatt

Haha, no problem!

Anton Ovcharenko

@anton-ovcharenko

Hi, is somewhere information about compatibility between the java version and jemalloc ver? Which is the best jemalloc candidate for openjdk version "11.0.10"? Thanks in advance!

Qi Wang

@interwq

We recommend the newest version. The only thing to make sure is, when using jemalloc 5.x, the JVM needs to include this fix: https://bugs.openjdk.java.net/browse/JDK-8215355
(It fixes a bug triggered often with jemalloc 5.x)

It looks the patch has been backported to JDK 11.0.7

Anton Ovcharenko

@anton-ovcharenko

thank you @interwq

Adam Lindley

@johadalin

Hi all. I'm trying to track down some native memory issues in a java app running on macOS (Big Sur). I was hoping that jemalloc could help, but while i can build it it doesn't seem to give me any heap profile info when i'm running.
I'm following https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md as a guide, replacing the .so with the .dylib mac produces.

Searching around, i see there's an old but still open issue suggesting that heap profiling just isn't implemented on MacOS: jemalloc/jemalloc#26 . Am I just out of luck here, and not going to be able to use jemalloc to dig into this? Or have i just failed to link it in properly at runtime or something? (Not sure how to check if the running java process picked up the linked library)

Dave Rigby

@daverigby

I’m reasonably sure that heap profiling still isn’t supported on macOS

I’ve not used it recently, but macOS Instruments used to have some good heap profiling support

Adam Lindley

@johadalin

:/ Well, i've got a bit further on loading the libraries. I was doing it wrong, but now have the right DYLD_INSERT_LIBRARIES env var set. Sadly, it's now failing to load them, because of signing issues. Apparently the jdk i'm running is a 'hardened process', and it complains

dyld: warning: could not load inserted library '/usr/local/lib/libjemalloc.dylib' into hardened process because no suitable image found.  Did find:
     /usr/local/lib/libjemalloc.dylib: code signature in (/usr/local/lib/libjemalloc.dylib) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)

after signing them with ldid -S.

Anyone got a clue where to get a non-hardened java runtime that i can use? Sadly all of this is not running in Xcode, because reasons, but i'll see if i can get macOS instruments going?

Dave Rigby

@daverigby

Worth during off System Integrity Protection (requires reboot into recovery mode)? That should disable all the signature nonsense ;)

https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection

Where communities thrive

People

Repo info

Activity