Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Brennan Vincent
    @umanwizard
    @interwq Thanks! Is the time of the official release still unknown?
    Qi Wang
    @interwq
    @umanwizard Sorry it’s been on our plan for a long time. I’ll try to get to it later this half. Best guess is by EOY.
    Brennan Vincent
    @umanwizard
    OK, thanks for the update.
    mayadav1
    @mayadav1
    Hi, Can anybody tell that jemalloc supports windows?
    Dave Rigby
    @daverigby
    @mayadav1 Yes, it does.
    agupta-zz
    @agupta-zz
    image.png
    Hi, mallctl call is failing with errno set to 2 (ENOENT). Could you please help with resolving the issue
    malloc.conf -> prof_active:false,prof_leak:true,lg_prof_sample:17,background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0
    David Goldblatt
    @davidtgoldblatt
    Probably you didn't pass --enable-prof to configure
    agupta-zz
    @agupta-zz
    i just used the default configuration. Any other configure option i should use ?
    David Goldblatt
    @davidtgoldblatt
    Nope; I think just prof is necessary for those mallctls to be available
    agupta-zz
    @agupta-zz
    alright. let me compile again and try. Thanks for your prompt response.
    David Goldblatt
    @davidtgoldblatt
    no problem!
    agupta-zz
    @agupta-zz
    @davidtgoldblatt i compiled with --enabled-prof. Still get the errno 2 for mallctl
    Am i missing anything ?
    Qi Wang
    @interwq
    @agupta-zz you will need to enable the prof at runtime too, e.g. by adding env var MALLOC_CONF=prof:true
    Vladimir
    @0xdeafbeef

    Can anybody help?

    This is with MALLOC_CONF="narenas:2,abort_conf:true,dirty_decay_ms:1000,muzzy_decay_ms:1000"
    INFO ton_indexer_test = RSS: 325.3 MB, Allocated: 305.6 MB, Metadata: 11.4 MB 
    INFO ton_indexer_test = Mapped: 334.0 MB, Active: 315.2 MB, Retained: 418.3 MB 
    
    This is with no configs at all.
    INFO ton_indexer_test_mi = RSS: 1.1 GB, Allocated: 456.3 MB, Metadata: 27.5 MB 
    INFO ton_indexer_test_mi = Mapped: 1.2 GB, Active: 496.9 MB, Retained: 2.3 GB

    It's rust application using rocks db. Both are using jemalloc.
    sizes from heaptrack

    Btw, mimalloc is using 2Gb and keep growing.

    Dave Rigby
    @daverigby
    What’s your question? why is the RSS and mapped bigger in second instance?
    Vladimir
    @0xdeafbeef
    Yes. And retained
    Dave Rigby
    @daverigby
    So for a start the default number of arens is something like 2x (4x?) number of logical cores, so assuming your machine has more than 1 core, you’ll have loads more arenas
    (and so if you’re allocating a range of sizes across many threads your heap could get pretty fragmented)
    I don’t know off the top of my head what the decay defaults are, but again I would suspect they are less aggressive than what you’re setting with that explciit config
    Vladimir
    @0xdeafbeef
    Basic decay is 10s, I have 12 cores
    Dave Rigby
    @daverigby
    ok, so you’ve got 48 areans there vs 2
    If you set opt.stats_print to true, you can see the state of the individual arenas at exit, that might help you understand the differences
    I suspect the difference in arena count will be the main reason - broadly it’s a performance vs space trade-off
    (space / fragmentation)
    Vladimir
    @0xdeafbeef

    What is the basic reason of fragmentation? Small values? Maybe there is a guide/doc for tuning it?
    I've just set

    narenas:2,abort_conf:true,dirty_decay_ms:10000,muzzy_decay_ms:10000

    and got

    INFO ton_indexer_test_mi = RSS: 1022.3 MB, Allocated: 501.1 MB, Metadata: 24.7 MB 
    INFO ton_indexer_test_mi = Mapped: 1.1 GB, Active: 526.2 MB, Retained: 974.3 MB

    So it's something with decay times

    Both RSS and retained are growing
    Dave Rigby
    @daverigby

    What is the basic reason of fragmentation? Small values?

    You want to allocate someting less than a page - e.g. 16B. jemalloc allocates memory using pre-defined sizes classes - i.e. all 16B allocations will come from a set of (4kB) pages; 32B allocations will come from a different set. That makes the maths to allocate / free allocations cheap and quick.

    But, that means that fragmentaion can be costly - e.g. first 16B allocation always costs 4KB, as you have to allocate first page
    Similarly, if you allocate enough 16B allocations to require 2 pages, but then free half of them at random, statisitcally you’ll still have half the allocaitons in each page, so your RSS is still 8KB
    multiply all that by the number of arenas, as each arena is independent...
    Re: the decay stuff, I’m not super familar on the details but that controls how often / aggressively jemalloc looks at the heap and tries to unmap (from the process address space) pages which are currently unused etc
    Dave Rigby
    @daverigby
    Dunno if there’s any newer writups, and certainly details have changed a bit, but https://engineering.fb.com/2011/01/03/core-data/scalable-memory-allocation-using-jemalloc/ is a reasonable overview
    Vladimir
    @0xdeafbeef
    Thank you!
    David Goldblatt
    @davidtgoldblatt
    https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md is the other thing we like to point people towards
    vinayhudli
    @vinayhudli
    Hi All, I stumbled upon jemalloc while trying to debug my java app for native memory leaks. I'm facing an issue when configuring jemalloc specifically 5.2.1 tag, when I issue : ./autogen.sh --enable-perf
    I get configure: WARNING: unrecognized options: --enable-perf .
    Appreciate any help
    David Goldblatt
    @davidtgoldblatt
    Hi; it's prof, not perf
    vinayhudli
    @vinayhudli
    yeah I just now figured that and realizing now that I should not have posted this. Thanks @davidtgoldblatt
    David Goldblatt
    @davidtgoldblatt
    Haha, no problem!
    Anton Ovcharenko
    @anton-ovcharenko
    Hi, is somewhere information about compatibility between the java version and jemalloc ver? Which is the best jemalloc candidate for openjdk version "11.0.10"? Thanks in advance!
    Qi Wang
    @interwq

    We recommend the newest version. The only thing to make sure is, when using jemalloc 5.x, the JVM needs to include this fix: https://bugs.openjdk.java.net/browse/JDK-8215355
    (It fixes a bug triggered often with jemalloc 5.x)

    It looks the patch has been backported to JDK 11.0.7

    Anton Ovcharenko
    @anton-ovcharenko
    thank you @interwq
    Adam Lindley
    @johadalin

    Hi all. I'm trying to track down some native memory issues in a java app running on macOS (Big Sur). I was hoping that jemalloc could help, but while i can build it it doesn't seem to give me any heap profile info when i'm running.
    I'm following https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md as a guide, replacing the .so with the .dylib mac produces.

    Searching around, i see there's an old but still open issue suggesting that heap profiling just isn't implemented on MacOS: jemalloc/jemalloc#26 . Am I just out of luck here, and not going to be able to use jemalloc to dig into this? Or have i just failed to link it in properly at runtime or something? (Not sure how to check if the running java process picked up the linked library)

    Dave Rigby
    @daverigby
    I’m reasonably sure that heap profiling still isn’t supported on macOS
    I’ve not used it recently, but macOS Instruments used to have some good heap profiling support
    Adam Lindley
    @johadalin

    :/ Well, i've got a bit further on loading the libraries. I was doing it wrong, but now have the right DYLD_INSERT_LIBRARIES env var set. Sadly, it's now failing to load them, because of signing issues. Apparently the jdk i'm running is a 'hardened process', and it complains

    dyld: warning: could not load inserted library '/usr/local/lib/libjemalloc.dylib' into hardened process because no suitable image found.  Did find:
         /usr/local/lib/libjemalloc.dylib: code signature in (/usr/local/lib/libjemalloc.dylib) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)

    after signing them with ldid -S.

    Anyone got a clue where to get a non-hardened java runtime that i can use? Sadly all of this is not running in Xcode, because reasons, but i'll see if i can get macOS instruments going?

    Dave Rigby
    @daverigby
    Worth during off System Integrity Protection (requires reboot into recovery mode)? That should disable all the signature nonsense ;)