Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    manveru
    @manveru:matrix.org
    [m]
    never would've thought this gets inherited into the sandbox
    Florian Apolloner
    @apollo13
    self._close_pipe_fds namely
    actually no, no loop in there
    wonder if I have any chance of getting that thing running
    manveru
    @manveru:matrix.org
    [m]
    maybe, if you have nix
    even then it's kinda hard
    Florian Apolloner
    @apollo13
    mhm, I got it to start (without the metric refreshing of course)
    but I still do not see it closing fds up to 1024 (which is my limit
    manveru
    @manveru:matrix.org
    [m]
    hmm, strange
    Florian Apolloner
    @apollo13
    I wonder if setting close_fds=False would fix it for you
    If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors.
    pretty sure that is your issue, let's see if I can repro
    manveru
    @manveru:matrix.org
    [m]
    hm, actually restic is written in Go and it also hung forever before i lowered the limits...
    Florian Apolloner
    @apollo13
    mhm, jeah that is it:
    strace -f python test.py 2>&1|grep close
    [pid 12769] close(66) = 0
    [pid 12769] close(67) = 0
    import subprocess
    import sys, os
    for i in range(100):
        os.dup2(1, i+10)
    subprocess.check_call(['ls', '/dev/fd'], close_fds=True)
    you can toggle close_fds there and it will perform better…
    that said, python should only close actually open file descriptors via /dev/fd in newer python versions
    manveru
    @manveru:matrix.org
    [m]
    thanks :)
    Florian Apolloner
    @apollo13
    not up to the rlimit, which python version do you have?
    without strace the program will output 0 1 2 for close_fds=True and otherwise up to 100
    109 actually
    on linux I nowhere (not even in the forked process) see a call to prlimit with RLIMIT_NOFILE
    manveru
    @manveru:matrix.org
    [m]
    the prlimit call was from calling lsof
    Florian Apolloner
    @apollo13
    ha I see it on python2.7
    prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=512*1024}) = 0
    and the close up to 1023
    with the test program I just sent
    manveru
    @manveru:matrix.org
    [m]
    this is running python 3.8.3... so not too old
    Florian Apolloner
    @apollo13
    oh
    sure about that :þ
    manveru
    @manveru:matrix.org
    [m]
    going to upgrade it to 3.8.7 now anyway
    Florian Apolloner
    @apollo13
    I mean then there might be a second issue somewhere else
    manveru
    @manveru:matrix.org
    [m]
    well, yes
    that python script calls lsof via subprocess :)
    Florian Apolloner
    @apollo13
    oh
    manveru
    @manveru:matrix.org
    [m]
    so whatever problems lsof causes, it'd inherit and show up on its strace
    Florian Apolloner
    @apollo13
    right, so anyways; python should just go through /dev/fd and close all open fds there if you have close_fds=True
    if the rest also breaks in lovely ways, that is on you :þ
    manveru
    @manveru:matrix.org
    [m]
    by the time i straced it, it was in a few billion entries... so had no chance of finding out what caused it
    then i tried debugging with lsof, and saw the same issues... didn't even think the script calls it ^^;
    Florian Apolloner
    @apollo13
    hehe great, either way the python subprocess module had a similar issue. so my debugging was not worthless at least :)
    manveru
    @manveru:matrix.org
    [m]
    anyway, thanks for the fact checking, this is a rabbit hole i've never been down :)
    Florian Apolloner
    @apollo13
    hehe, no worries; I learned a bit
    gotta run, good luck
    manveru
    @manveru:matrix.org
    [m]
    👋