Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Radostin Stoyanov
    @rst0git
    @Snorch I am able to replicate the compilation errors from CI with fedora:rawhide container.
    Radostin Stoyanov
    @rst0git
    I've opened a pull request with a fix: checkpoint-restore/criu#1962
    Bui Quang Minh
    @minhbq-99
    Hi everyone, I'm trying to implement C/R support for cgroupv2 threaded controller which means threads in a process may belong to different controllers.
    As threads are cloned and restored later in restorer, my idea is to create a service fd (cgroupd) working like usernsd that receives the cg_set number from restored thread and the thread id then fix up the thread's controller (write thread id to controller/cgroup.threads). However, AFAIK, SCM_CREDENTIALS cmsg contains the process id (thread group id) not the thread id. So how can we pass the thread id across the namespace boundary?
    Pavel Tikhomirov
    @Snorch

    my idea is to create a service fd (cgroupd) working like usernsd

    Why not just use usernsd, e.g. see how userns_move works, but just give the tid in it's arguments? (instead of using the one SCM_CREDENTIALS give you)

    Bui Quang Minh
    @minhbq-99
    I think the reason behind using SCM_CREDENTIALS is that it transforms the pid of caller (which may be in pid namespace) into outer pid namespace of callee (usernsd). If we pass tid directly from the inside pid namespace, it may be not correct tid from the usernsd outer pid namespace viewpoint.
    Pavel Tikhomirov
    @Snorch
    Just send item->threads[i].real as usernsd should be in criu pidns. upd: this is probably unavailable on restore, but it should not be too hard to get it from proc.
    Bui Quang Minh
    @minhbq-99
    Thanks, I will look around and try to tackle that
    Younes Manton
    @ymanton
    Is it possible for a test program to know where the parasite blob lives? I'm trying to write a test that checks the parasite blob's stack, but I don't see any existing way to do that. Maybe the test has to dig around its proc, but I was hoping a nicer way existed
    SnaK
    @SallyKAN

    Hi everyone, I am trying to checkpoint a process, but I got this error message:

    [ff.checkpoint] (0.085s) criu> (00.035845) ----------------------------------------
    [ff.checkpoint] (0.085s) criu> (00.036034)
    [ff.checkpoint] (0.085s) criu> (00.036039) Dumping pages (type: 58 pid: 1000)
    [ff.checkpoint] (0.085s) criu> (00.036041) ----------------------------------------
    [ff.checkpoint] (0.085s) criu> (00.036772) Pagemap generated: 1792 pages (0 lazy) 0 holes
    [ff.checkpoint] (0.085s) criu> (00.039268) Error (criu/page-xfer.c:254): page-xfer: Unable to spice data: Broken pipe
    [ff.checkpoint] (0.085s) criu> (00.039285) Error (criu/bfd.c:132): bfd: Error flushing image: Broken pipe
    [ff.checkpoint] (0.085s) criu> (00.039361) ----------------------------------------
    [ff.checkpoint] (0.085s) criu> (00.039364) Error (criu/mem.c:644): Can't dump page with parasite
    [ff.checkpoint] (0.085s) criu> (00.039377) Error (criu/bfd.c:132): bfd: Error flushing image: Broken pipe
    [ff.checkpoint] (0.085s) criu> (00.041419) Error (criu/bfd.c:132): bfd: Error flushing image: Broken pipe
    [ff.checkpoint] (0.085s) criu> (00.041458) Unlock network
    [ff.checkpoint] (0.085s) criu> (00.041462) Unfreezing tasks into 1
    [ff.checkpoint] (0.085s) criu> (00.041547) Dismissing the image streamer
    [ff.checkpoint] (0.085s) criu> (00.041558) Error (criu/cr-dump.c:1792): Dumping FAILED.

    Also, I read about these explaining how CRIU implements checkpoint TCP connection (https://criu.org/TCP_connection), but it's more about how they deal with sockets during the restore process.
    Can someone help to point me out why this error happens during the checkpoint process? ? Thanks in advance!

    I am also wondering can CRIU checkpoint a TCP socket with the Keep-Alive option?
    Adrian Reber
    @adrian:lisas.de
    [m]
    @SallyKAN: the output looks unusual. How are you using CRIU?
    SnaK
    @SallyKAN
    well this is actually a open source tool called fastfreeze which assemble the CR function of CRIU, it will prints the error log of CRIU execution
    it seems like CRIU dumping task failed because of a tcp socket... when I removed the tcp connection, it works fine...
    Adrian Reber
    @adrian:lisas.de
    [m]
    @SallyKAN: maybe try it first without fast freeze. @nviennot is the author of fast freeze and is also reachable here (sometimes)
    SnaK
    @SallyKAN
    I don't quite understand here in the source code of CRIU, it seems like the splice method needs to write to the sockets of the processs I am gonna checkpoint.
    /* local xfer */
    static int write_pages_loc(struct page_xfer *xfer, int p, unsigned long len)
    {
        ssize_t ret;
        ssize_t curr = 0;
    
        while (1) {
            ret = splice(p, NULL, img_raw_fd(xfer->pi), NULL, len - curr, SPLICE_F_MOVE);
            if (ret == -1) {
                pr_perror("Unable to spice data");
                return -1;
            }
            if (ret == 0) {
                pr_err("A pipe was closed unexpectedly\n");
                return -1;
            }
            curr += ret;
            if (curr == len)
                break;
        }
    
        return 0;
    }

    @SallyKAN: maybe try it first without fast freeze. @nviennot is the author of fast freeze and is also reachable here (sometimes)

    Thanks for your replying!

    Pavel Tikhomirov
    @Snorch

    I don't quite understand here in the source code of CRIU, it seems like the splice method needs to write to the sockets of the processs I am gonna checkpoint.

    In the above code splice writes memory of your dumped processes to image files.

    SnaK
    @SallyKAN

    I don't quite understand here in the source code of CRIU, it seems like the splice method needs to write to the sockets of the processs I am gonna checkpoint.

    In the above code splice writes memory of your dumped processes to image files.

    So does this mean that splice will read all the fds of my process first? I am trying to figure out in what circumstances it will throw the page-xfer: Unable to spice data: Broken pipeerror

    Pavel Tikhomirov
    @Snorch

    You can use https://github.com/Snorch/linux-helpers/blob/master/gftrace.sh like this

    perf probe -f '__x64_sys_splice%return $retval'
    bash ./gftrace.sh __x64_sys_splice

    And reproduce the problem while script is running.

    Depending on your kernel the exact traceable name of sys_splice may be different. Search for it in /sys/kernel/debug/tracing/available_filter_functions if needed.

    And provide output file (./trace), that would probably shed more light on what happens in your case.

    Normally EPIPE is returned if other end of pipe is closed and thus we would never be able to get/send data from/to it.
    Alternatively there can be other error messages in criu log, and EPIPE is not actual problem, so please attach full criu log, else it is hard to help.
    Zeyad Yasser
    @ZeyadYasser

    Hello Everyone,
    How do I make CRIU dump opened files that are on tmpfs or dev mounts (e.g. /tmp, /dev/shm). I know CRIU supports this, but I can't get it work.

        16: Error (criu/files-reg.c:2259): Can't open file dev/shm/mono.16 on restore: No such file or directory
        16: Error (criu/files-reg.c:2185): Can't open file dev/shm/mono.16: No such file or directory
        16: Error (criu/mem.c:1359): `- Can't open vma
        15: Error (criu/cr-restore.c:1494): 16 exited, status=1

    I tried specifying --external for those mountpoints, but still they are not being dumped.
    Thanks!

    Adrian Reber
    @adrian:lisas.de
    [m]
    If you say external it will definitely not work. Good question. Not entirely sure, but maybe CRIU only dumps a tmpfs if you are running in a mount namespace. Not sure. It works always for containers. @Snorch do you know when CRIU also includes the tmpfs contents in the checkpoint?
    Radostin Stoyanov
    @rst0git

    How do I make CRIU dump opened files that are on tmpfs or dev mounts (e.g. /tmp, /dev/shm)

    There are a few ways of doing this. It depends on your use case. For instance, you can use action-script as shown in the following example.
    https://github.com/checkpoint-restore/criu/blob/criu-dev/scripts/tmp-files.sh

    Radostin Stoyanov
    @rst0git

    Can't open file dev/shm/mono.16

    btw, we recently added support in Podman to checkpoint/restore the content of dev/shm: containers/podman#12665

    Pavel Tikhomirov
    @Snorch
    @ZeyadYasser
    1) CRIU dumps mounts only when dumped process mount namespace is dumped
    2) Mount namespace is dumped if process is in different mount namespace to CRIU (CRIU assumes mount namespace belongs to dumped process exclusively like in a container)
    3) When tmpfs mount is dumped its content is always collected in tar image and on restore it is put back in newly created tmpfs
    4) External mounts for CRIU is kind of a blackbox, CRIU does not dump them, the user should provide all needed mounts with exactly the same content on restore via CRIU options.
    Zeyad Yasser
    @ZeyadYasser
    Thanks everyone, it makes much more sense now.
    s09bQ5
    @s09bQ5
    Hi, am I right that CRIU is able to handle System V message queues but doesn't know how to handle POSIX message queues? For me it chokes on the file descriptors of POSIX message queues that exist on an invisible mount. Or do I have to mount the mqueue filesystem in a special way?
    3 replies
    Vaibhav Jakkula
    @VaibhavJak
    Hi all . I am a 2nd year BTech(CSAI) student of IIIT Lucknow. Interested in docker and linux. Please suggest me some of the relevant projects and how to get started to contributing here, will be highly appreciated:)
    Bui Quang Minh
    @minhbq-99
    @VaibhavJak You can get some project ideas from issues or this page: https://criu.org/Google_Summer_of_Code_Ideas (some are in developing progress you may check the pull request list)
    Bui Quang Minh
    @minhbq-99

    Hi everyone, I am debugging a failed test in my cgroup-v2 PR. In global properties, we use access(path, F_OK) if errno == ENOENT, we simply skip this global property. However, I observe that in the failed test the errno of access(path, F_OK) is EACCESS but when opening the file the errno is ENOENT which is weird to me. I also see some access(path, F_OK) returns EACCESS but can still open and read that file.
    My patch for getting some logs

    diff --git a/criu/cgroup.c b/criu/cgroup.c
    index 2cdb63609..f4f50fd38 100644
    --- a/criu/cgroup.c
    +++ b/criu/cgroup.c
    @@ -382,14 +382,19 @@ static int dump_cg_props_array(const char *fpath, struct cgroup_dir *ncd, const
            struct cgroup_prop *prop;
    
            for (j = 0; cgp && j < cgp->nr_props; j++) {
    +               int ret;
    +
                    if (snprintf(buf, PATH_MAX, "%s/%s", fpath, cgp->props[j]) >= PATH_MAX) {
                            pr_err("snprintf output was truncated\n");
                            return -1;
                    }
    
    -               if (access(buf, F_OK) < 0 && errno == ENOENT) {
    +               ret = access(buf, F_OK);
    +               if (ret < 0 && errno == ENOENT) {
                            pr_info("Couldn't open %s. This cgroup property may not exist on this kernel\n", buf);
                            continue;
    +               } else if (ret < 0) {
    +                       pr_perror("cgroup: Path: %s", buf);
                    }
    
                    prop = create_cgroup_prop(cgp->props[j]);

    Error log

    (00.071032) Error (criu/cgroup.c:397): cg: cgroup: Path: /proc/self/fd/16/bar/cgroup.clone_children: Permission denied
    (00.071045) cg: Dumping value 0 from /proc/self/fd/16/bar/cgroup.clone_children
    (00.071049) Error (criu/cgroup.c:397): cg: cgroup: Path: /proc/self/fd/16/bar/notify_on_release: Permission denied
    (00.071056) cg: Dumping value 0 from /proc/self/fd/16/bar/notify_on_release
    (00.071060) Error (criu/cgroup.c:397): cg: cgroup: Path: /proc/self/fd/16/bar/cgroup.procs: Permission denied
    (00.071066) cg: Dumping value  from /proc/self/fd/16/bar/cgroup.procs
    (00.071069) Error (criu/cgroup.c:397): cg: cgroup: Path: /proc/self/fd/16/bar/tasks: Permission denied
    (00.071075) cg: Dumping value  from /proc/self/fd/16/bar/tasks
    (00.071079) Error (criu/cgroup.c:397): cg: cgroup: Path: /proc/self/fd/16/bar/cgroup.subtree_control: Permission denied
    (00.071084) Error (criu/cgroup.c:292): cg: Failed opening /proc/self/fd/16/bar/cgroup.subtree_control: No such file or directory
    (00.071085) Error (criu/cgroup.c:462): cg: dumping global properties failed
    (00.071088) Error (criu/cgroup.c:732): cg: failed walking /proc/self/fd/16/ for empty cgroups: No such file or directory
    Bui Quang Minh
    @minhbq-99
    Hi everyone, this is not related to CRIU but what tools do you usually use for getting stack trace, call graph like ftrace but in the userspace?
    Pavel Tikhomirov
    @Snorch
    Gdb, strace and probably eBPF
    Bui Quang Minh
    @minhbq-99
    Thanks @Snorch , forgot to mention I want to get trace of userspace function, strace is used to get system call trace only afaik. I search more about eBPF tools and find out uprobe, let me give it a try.
    Adrian Reber
    @adrian:lisas.de
    [m]
    @Snorch: I am currently trying some changes in CRI-O for Kubernetes and I get:
    (00.029005)      1: Error (criu/mount-v2.c:891): mnt-v2: Failed to copy sharing from -1:/var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv to 11: Invalid argument
    (00.029028)      1: Error (criu/mount-v2.c:958): mnt-v2: Failed to copy sharing from source /var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv to 656
    (00.029821) Error (criu/mount.c:3674): mnt: Can't remove the directory /tmp/.criu.mntns.kzWYEa: Device or resource busy
    (00.029843) Error (criu/cr-restore.c:2536): Restoring FAILED.
    If I switch to compat mode (no v2 mount) the restore works
    Not sure if I am doing something wrong
    any ideas
    Pavel Tikhomirov
    @Snorch
    I will look on it, would be nice to have a reproduce
    Pavel Tikhomirov
    @Snorch

    @adrian:lisas.de https://github.com/Snorch/linux-helpers/blob/master/gftrace.sh collecting gftrace would be helpful

    perf probe -f 'do_set_group%return $retval'
    ./gftrace do_set_group

    at the time of error.

    most likely one of the conditions in do_set_group resulting with einval fails
    Younes Manton
    @ymanton
    How should socket options be handled across kernels? In 3.17 we started dumping SO_BUF_LOCK, but if you checkpoint on a kernel that has that and restore on a kernel that doesn't it fails with ENOPROTOOPT. Maybe we should be checking the kerndat for support on the restore side as well for such options? We can't really fix it properly if a kernel doesn't have that option.
    Pavel Tikhomirov
    @Snorch
    I believe general design is to fail if something dumped can’t be restored. One option here is to use crit decode/encode to modify images to work around your case.
    Younes Manton
    @ymanton
    Thanks, I'll look into that.
    SnaK
    @SallyKAN

    Hey everyone, I have a question about the compatibility of CPUID instruction in CRIU. Recently I have been researching the work that Fastfreeze did, it mentioned some scenarios, like the one below:

    Suppose an application is started on a host which supports the AVX instruction set. On startup, the application detects the AVX capability by invoking the CPUID instruction and chooses to use AVX-enabled versions of certain functions, such as memcpy. If the running application is migrated to a non-AVX capable host, the application, oblivious to the migration, keeps running its AVX code paths, and crashes with a SIGILL soon after.

    And Fastfreeze implements libvirtcpuid to hide AVX feature during application startup, I am wondering if CRIU has some mechanisms to deal this? Thanks in advance.

    James Yang
    @futurist

    I'm using runc 1.1.4(on debian11) to checkpoint a nginx container but failed, the error is:

    (02.530007) 4562 fdinfo 0: pos:                0 flags:           102002/0
    (02.530041) tty: Dumping tty 17 with id 0xd
    (02.530047) Error (criu/files-reg.c:1710): Can't lookup mount=6219 for fd=0 path=/dev/pts/0
    (02.530067) ----------------------------------------
    (02.530080) Error (criu/cr-dump.c:1635): Dump files (pid: 4562) failed with -1
    (02.530089) Waiting for 4562 to trap
    (02.530094) Daemon 4562 exited trapping

    The pid 4562 is the nginx master process. Can anyone help debug with this?

    Adrian Reber
    @adrian:lisas.de
    [m]
    @futurist: how did you start the container? Exact command line if possible
    Yixue Zhao
    @felicitia

    Hello everyone! I'm wondering if there are any plans for porting criu to RISC-V? The closest effort I found is this Git issue (checkpoint-restore/criu#1702) and I already talked to Rushi.

    I'd love to get an estimate on how much effort is needed, and would love to help! but I'm new to criu and system programming so I'm wondering if there's anyone who's more experienced that I can talk to? Thanks a lot!

    Alexander Mikhalitsyn
    @mihalicyn
    Hello, yep, this is a really interesting project. I've it on my ToDo. But I believe that it's required to have the hardware to work on that [I personally, haven't bought it yet. This is a blocker for me :-)]
    It's a real pain to compile something in a fully-emulated environment. I have got an experience with MIPS emulation and CRIU compilation took ~5 min on my 12-thread CPU (you can use cross-compilation, but... anyway)