Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    David Goldblatt
    @davidtgoldblatt
    Well you could imagine a situation where pointers get moved from one copy of the library to another
    and that would be bad
    i.e. alloc with libjemalloc_lib1.so, free with libjemalloc_lib2.so or something
    Coleman
    @cenomla
    makes sense
    The library that is causing the crash is libGLESv2 so it's not like I have a lot of control over what it links with
    I'm trying to get a wrap.sh file setup rn so maybe I can do some ld preloading magic
    Ed
    @edtbl76
    Any documentation whatsoever on using this as a profiler?
    David Goldblatt
    @davidtgoldblatt
    https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling ?
    Or the man page
    Ed
    @edtbl76
    I've gone through both. I guess going through the src code is the next best bet. It would be awesome if it was slightly better documented.
    Renaud Lepage
    @cybik
    dunno if anyone's around, but if you are: I've been experimenting with jemalloc, and am seeing way different behaviour (expectedly, to be fair) between 3.6.0 and HEAD. if somebody knows why that is off the top of their head before I dive further into bisects and research and the such, it'd be appreciated.
    Qi Wang
    @interwq
    The different behaviour is expected indeed; for example we had an internal metadata redesign with 5.0; the time based decay also changes the purging behavior completely. Which aspect you are looking at? Or any changes to the wrong direction?
    Renaud Lepage
    @cybik
    specifically: I shoved jemalloc in a game through LD_PRELOAD and one of my worst memory offenders (35%+ on a 32gbRAM system) went straight the hell down to 8%
    HEAD goes back to 30% except it also goes down to <10% if and when it so pleases (a.k.a. some specific threshold)
    I did see purge:decay and I want to try setting it to purge:ratio again :P
    unless I can do purge:decay,ratio
    Qi Wang
    @interwq
    That sounds like the time based decay; see http://jemalloc.net/jemalloc.3.html#opt.dirty_decay_ms
    Ratio based decay is removed; sorry :) in our experience the time based approach performs almost always better. It may cache pages longer, but usually at the benefit of CPU wins.
    Renaud Lepage
    @cybik
    nooooooooooooooooooooooooooooo :(
    also my issue is that I play with memory across threads, which made je 3.6.0 kinda perfect
    any advice on "restoring" (or ways of imitating) 3.6's behaviour?
    Renaud Lepage
    @cybik
    ...oddly enough, only the ubuntu-provided jemalloc displays the "magic behaviour"
    Qi Wang
    @interwq
    if you can share the malloc_stats with the 2 different versions, we can take a look.
    It might be the default purging settings. With 5.x, adding dirty_decay_ms:0 will purge dirty pages immediately. See https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md
    Renaud Lepage
    @cybik
    at compile time, I would assume?
    Oh /redacted/ YES.
    git clean -fdx; ./autogen.sh --enable-static --enable-shared --prefix=`pwd`/../5.2.1/x64_devbox_v3 --disable-prof --disable-stats --with-malloc-conf=dirty_decay_ms:0 --with-lg-page=12 --with-lg-hugepage=21 && make -j9 && make install makes the magic happen
    Renaud Lepage
    @cybik
    all right so my stuff still peaks at 31% then goes straight the hell down to 2%. Not sure how I'm hitting precisely that point before nosediving or whether that threshold means something specific
    Qi Wang
    @interwq
    The decay part is likely the key. The eager decay purges memory fast, however it may does more syscalls. The peak usage is probably not related to the setting changes here. Does the other version show similar peak?
    Renaud Lepage
    @cybik
    3.6.0 did show similar peaks, I believe.
    i can do a quick check, hold pls
    yep, peaks around the same point then frees memory like it needed to kill it with intent.
    I can also say that jemalloc Gen5 is actually giving better memory results than the height of Gen3 - congrats on that
    Qi Wang
    @interwq
    Thanks :) let us know if any questions.
    Renaud Lepage
    @cybik
    honestly the only remaining one is whether you have any tips on finding peak usage sources. I'm betting that's what jeprof is for though :3
    David Goldblatt
    @davidtgoldblatt
    BTW, you don't need to recompile to change purging settings; --with-malloc-conf will set the default values, but you can also change them via the MALLOC_CONF environment variable
    (or a couple of other ways, including programmatically; the man page has the details)
    Renaud Lepage
    @cybik
    hopefully i can nuke that conf method for the final build of the thing I'm making
    David Goldblatt
    @davidtgoldblatt
    And yeah, jeprof should help in tracking down allocation traces responsible for the most live memory
    Renaud Lepage
    @cybik

    O.o

    so uh looks like jemalloc doesn't play nice with steam's drm wrapping. is that known?

    David Goldblatt
    @davidtgoldblatt
    Nope
    Does steam interpose malloc or something?
    Renaud Lepage
    @cybik
    hopefully not
    David Goldblatt
    @davidtgoldblatt
    I don't see how they could be affected
    Renaud Lepage
    @cybik
    well the only thing that changed between my running that exact executable locally, and it being uploaded, is drmwrapping
    [285195.922549] traps: Indivisible.x86[18378] general protection ip:55dd3b29a7c2 sp:7ffea8f850d0 error:51c0 in Indivisible.x86_64-pc-linux-gnu[55dd3aca0000+8c4000] [285205.441425] traps: Indivisible.x86[18388] general protection ip:556afec707c2 sp:7ffd6aa92880 error:f1c0 in Indivisible.x86_64-pc-linux-gnu[556afe676000+8c4000] [285223.792690] traps: Indivisible.x86[18395] general protection ip:55aca774c7c2 sp:7ffd03e0f760 error:21c0 in Indivisible.x86_64-pc-linux-gnu[55aca7152000+8c4000] [285255.329130] traps: Indivisible.x86[18516] general protection ip:55dd9692d7c2 sp:7ffd4896f410 error:80d0 in Indivisible.x86_64-pc-linux-gnu[55dd96333000+8c4000] [285258.527824] traps: Indivisible.x86[18526] general protection ip:55bd8601a7c2 sp:7ffdf5205e40 error:70d0 in Indivisible.x86_64-pc-linux-gnu[55bd85a20000+8c4000] [285259.592341] traps: Indivisible.x86[18535] general protection ip:56387f9517c2 sp:7ffe3c377540 error:40d0 in Indivisible.x86_64-pc-linux-gnu[56387f357000+8c4000] [285260.440867] traps: Indivisible.x86[18544] general protection ip:55d5135857c2 sp:7fff914c7530 error:c0d0 in Indivisible.x86_64-pc-linux-gnu[55d512f8b000+8c4000] [285261.210894] traps: Indivisible.x86[18553] general protection ip:5642db1a47c2 sp:7fff002a21f0 error:60d0 in Indivisible.x86_64-pc-linux-gnu[5642dabaa000+8c4000] [285261.952536] traps: Indivisible.x86[18562] general protection ip:55954667e7c2 sp:7ffd5d77e6a0 error:d0 in Indivisible.x86_64-pc-linux-gnu[559546084000+8c4000] [285304.519802] traps: Indivisible.x86[18608] general protection ip:55a4e08347c2 sp:7ffd7d710160 error:a0d0 in Indivisible.x86_64-pc-linux-gnu[55a4e023a000+8c4000]
    sorry for the jumbled mess
    David Goldblatt
    @davidtgoldblatt
    No idea off the top of my head
    Renaud Lepage
    @cybik
    ...steam obfuscates the executable by default.
    yep, jemalloc and Linux -> can't obfuscate the executable