Dear all, finally we managed to solve the issue using Rancher with the following workaround extracted from coment of gitlawr on rancher/rancher#14836 . Literally we did the following steps
1) Edit cluster, Edit as YAML
2) Add the following flags for kubelet:
services:
kubelet:
extra_args:
containerized: "true"
extra_binds:
- "/:/rootfs:rshared"
3) Click save and wait till the cluster is updated.Notes:
The community is planning to deprecate the "--containerized" for kubelet(kubernetes/kubernetes#74148).
But the flag is essential for the capability as there is no alternative at the moment.
Now our REANA deployment on Rancher works! Yihaaa!
Thanks for the help and pointing the link!
The reason why I need this is because I have a lot of big files (GB) as input and I don’t want to replicate this files for each user’s workflow.
reana-workflow-engine-snakemake in the near future, but if there is a momentum and some people willing to contribute, we can perhaps revive this?!
@mdehollander Snakemake is actually something that was on our eyes since some time, e.g.it is used by some LHCb physics group as well. There are two possible ways: (1) running Snakemake workflows via CWL; (2) running Snakemake workflows directly. Ad (1), we have tested Snakemake -> CWL export -> REANA in the past, with mixed results. A direct support would be probably more advantagous, but we can retest that with latest Snakemake/CML versions. Ad (2), we have no immediate plans to develop
reana-workflow-engine-snakemakein the near future, but if there is a momentum and some people willing to contribute, we can perhaps revive this?!
@tiborsimko Thanks for your answer. I think indeed direct support for Snakemake is better, because I see that CWL export has quite some limitations: https://snakemake.readthedocs.io/en/stable/executing/interoperability.html#cwl-export I could see if there is interest in the snakemake community. I use snakemake quite often myself. But first I need to see if REANA would be of added value for us. From the website/paper it is, but a hands-on experience works always better. I could setup a local installation, but if there is a demo login I am happy to hear about it. For our end-users, the REANA web-interface would be also a good feature. Is there already a place where I can see how the ui will look it?
- We don't have any publicly accessible REANA cluster that you could use, although we were musing about setting one up; it would require some CPU accounting limits etc which we did not get to doing yet.
- The web interface is something that we are actively developing these months, so while there is
reana-uicomponent where you could see some mock-ups, it is better to wait for a more complete "live" version coming in February. We have user logins connected to GitLab for easy Continuous Integration use cases, we are working on workflow list and workflow run details visualisation. The important design principle is doing heavy stuff as React components, so that the UI could be eventually plugged into JupyterLab notebooks, so that one could easily launch workflows from within notebooks and monitor progress, for example.
Our main computing facility is the NERSC Cori machine at Berkeley National Lab (which runs SLURM for compute nodes, shifter for containers, and a separate k8 service called Spin), has anyone already successfully setup REANA there? And if not, any recommendations for how to go about it?