actually no, no loop in there
wonder if I have any chance of getting that thing running
even then it's kinda hard
but I still do not see it closing fds up to 1024 (which is my limit
I wonder if setting close_fds=False would fix it for you
If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors.
as an argument to popen and friends: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
pretty sure that is your issue, let's see if I can repro
hm, actually restic is written in Go and it also hung forever before i lowered the limits...
strace -f python test.py 2>&1|grep close
[pid 12769] close(66) = 0
[pid 12769] close(67) = 0
[pid 12769] close(67) = 0
import subprocess
import sys, os
for i in range(100):
os.dup2(1, i+10)
subprocess.check_call(['ls', '/dev/fd'], close_fds=True)
you can toggle close_fds there and it will perform better…
that said, python should only close actually open file descriptors via /dev/fd in newer python versions
without strace the program will output 0 1 2 for close_fds=True and otherwise up to 100
109 actually
on linux I nowhere (not even in the forked process) see a call to prlimit with RLIMIT_NOFILE
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=512*1024}) = 0
and the close up to 1023
with the test program I just sent
sure about that :þ
that python script calls
lsof via subprocess :)
so whatever problems lsof causes, it'd inherit and show up on its strace
if the rest also breaks in lovely ways, that is on you :þ
by the time i straced it, it was in a few billion entries... so had no chance of finding out what caused it
then i tried debugging with lsof, and saw the same issues... didn't even think the script calls it ^^;
anyway, thanks for the fact checking, this is a rabbit hole i've never been down :)
gotta run, good luck