Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    David Goldblatt
    @davidtgoldblatt
    What do you mean by grow_retain?
    Qi Wang
    @interwq
    likely during extent_grow_retained, we have error paths
    in which case may trigger dalloc / merge
    which calls down the try_acquire function newly added
    David Goldblatt
    @davidtgoldblatt
    Right, sorry, all I mean is, currently we have lines like
        witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn),
            WITNESS_RANK_CORE, growing_retained ? 1 : 0);
    And they should really be
    Qi Wang
    @interwq
    Ah yes
    Right that should be an exact rank
    David Goldblatt
    @davidtgoldblatt
    if (growing_retained) {
        witness_assert_lock_held_at_rank(..., WITNESS_RANK_GROW_MTX);
    }
    Qi Wang
    @interwq
    Agreed.
    David Goldblatt
    @davidtgoldblatt
    idk, I mean, I'd be OK if this was scope creeping too much; I'm fine with anything reasonable
    I was just confused at the name, because when I saw "min depth" I got confused and looked for the depth
    Qi Wang
    @interwq
    I now find it confusing too.
    How about, we stick with postive_depth
    which don’t take a depth parameter
    David Goldblatt
    @davidtgoldblatt
    Makes sense to me
    Qi Wang
    @interwq
    i.e. witness_assert_positive_depth_to_rank
    Okay I’ll update
    vient
    @vient
    Hi! What is the best way to get callbacks similar to malloc_hook? I just want to know if malloc was called and extent hooks require to implement some logic. I also saw junk_alloc_callback, is it safe to use it? It is defined in some internal header after all.
    David Goldblatt
    @davidtgoldblatt
    We don't have a great analogue (junk_alloc_callback isn't a great fit)
    We have experimental functionality that's equivalent (the experimental.hooks.install and experimental.hooks.remove mallctl calls)
    But it's not "stable", and has a high likelihood of getting removed
    vient
    @vient
    That's unfortunate. Just in case, my problem is that I want to know if malloc or free were called during execution of a test. Do you know if there are any "good" ways to solve this while using jemalloc, if callbacks are not an option as you say?
    About junk_alloc_callback, is it just "not a good fit" or not usable for this problem at all? Is it not stable? I'm fine with temporarily using new arena for a test with all needed parameters set for callbacks to work but stability of approach is important for me.
    David Goldblatt
    @davidtgoldblatt
    So, junk_alloc_callback doesn't get exposed externally; it's occasionally visible externally depending on exact build settings, but it's only really there for jemalloc unit tests
    I'm not sure if there's a good way to solve this with jemalloc
    It's not necessarily implementable without cost to the common case where the hook is disabled, so we've been very skeptical of comitting to it
    vient
    @vient
    I think callback costs are unavoidable without some kind of runtime patching
    Thanks for answering, I'll try to look into symbol wrapping then.
    David Goldblatt
    @davidtgoldblatt
    So, the trick we do with a lot of these is to fold it into a slow path; e.g. we have one initial branch of "will this allocation push us over the limit for bytes-allocated-before-we-do-some-background-work"
    And then, go to a slow path where we branch on more things
    So we do that here, too, with the experimental hooks
    Even the slow path isn't that slow, and the more we add to it the slower it gets
    So there's sort of a fundamental tradeoff that we haven't really decided
    s1341
    @s1341
    hi
    So I'm trying to use the jemallocator rust crate on aarch64-android. (https://github.com/gnzlbg/jemallocator). But I'm getting the following build error:`
     ../jemalloc/src/pages.c:534:11: error: reference to overloaded function could not be resolved; did you mean to call it?
              int fd = open("/sys/kernel/mm/transparent_hugepage/enabled", O_RDONLY);
                       ^~~~
      ../jemalloc/include/jemalloc/internal/hooks.h:9:28: note: expanded from macro 'open'
      #define open JEMALLOC_HOOK(open, hooks_libc_hook)
                                 ^~~~
      ../jemalloc/include/jemalloc/internal/hooks.h:7:71: note: expanded from macro 'JEMALLOC_HOOK'
      #define JEMALLOC_HOOK(fn, hook) ((void)(hook != NULL && (hook(), 0)), fn)
                                                                            ^~
      /nix/store/j820166rvxxraf006p2gxrq4g8gyldxx-bionic-prebuilt-ndk-release-r23-dev/include/bits/fortify/fcntl.h:70:5: note: possible target for call
      int open(const char* const __pass_object_size pathname, int flags, mode_t modes)
          ^
      /nix/store/j820166rvxxraf006p2gxrq4g8gyldxx-bionic-prebuilt-ndk-release-r23-dev/include/bits/fortify/fcntl.h:59:5: note: possible target for call
      int open(const char* const __pass_object_size pathname, int flags)
          ^
      /nix/store/j820166rvxxraf006p2gxrq4g8gyldxx-bionic-prebuilt-ndk-release-r23-dev/include/bits/fortify/fcntl.h:49:5: note: possible target for call
      int open(const char* pathname, int flags, mode_t modes, ...) __overloadable
          ^
      /nix/store/j820166rvxxraf006p2gxrq4g8gyldxx-bionic-prebuilt-ndk-release-r23-dev/include/fcntl.h:136:5: note: possible target for call
      int open(const char* __path, int __flags, ...);
          ^
      1 error generated.
    anyone know I can fix this?
    s1341
    @s1341
    ok. i just checked. I get the same error on jemalloc git dev branch.
    s1341
    @s1341
    i've opened an issue on github...
    Siddharth Likhate
    @slikhate_gitlab
    Hi All, We are considering replacing the default allocator on Windows and Linux with jemalloc. Initial tests have been very promising and show performance gains and reduced memory usage. As part of this potential allocator replacement, we are doing a security assessment of the jemalloc allocator. There are articles/papers that describe heap exploitation techniques for the default system allocators on Windows/Linux and for the Jemalloc allocator. However, there isn't really concrete info on comparing the allocators in terms of security. We are trying to understand what (if anything) we would be losing in terms of security when we switch from the default allocators to Jemalloc. Does anyone have guidance/pointers that they could share? Have other folks done this kind of security assessment when replacing their current allocators with Jemalloc?
    David Goldblatt
    @davidtgoldblatt
    I don't know of any systematic comparison. But I've chatted once or twice with the Windows malloc dev team, and I'd say that there's a difference of approach. Our default answer in common configurations to "should we be slower to increase security" is "no", and theirs is "it depends".
    (Also, fair warning: Windows support is something of a second class citizen; we make sure it works and passes tests, but none of the core team is highly experienced with the windows dev environment)
    Siddharth Likhate
    @slikhate_gitlab
    Thanks @davidtgoldblatt for the prompt response! Any comments/thoughts around the security of the Linux glibc malloc v/s Jemalloc? Also, out of curiosity, how often do you folks hear of security vulnerabilities reported against Jemalloc and what is the overall thought process around addressing them?
    David Goldblatt
    @davidtgoldblatt
    Mixed, I'd say; they care more, but also do embedded metadata which makes it easier to turn a buffer overrun into an attacker-controlled-write primitive (I think this has gotten better in recent glibcs? not sure)
    I don't know that we've ever seen evidence of a security vulnerability within jemalloc (as opposed to use-after-free or whatever vulnerabilities induced by applications that happen to be using jemalloc)
    We'll cut a new release if we find and fix a correctness bug
    Thinking back, there's only one bug fix I can think of that could plausibly be a security exploit, and that required several unusual properties of the user program (using a custom extension API, having a set of page hook replacements which don't zero new memory, and relying on this in the program via e.g. calloc calls)