Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 12 08:03
    mvidalgarcia closed #472
  • Jan 11 18:25
    mvidalgarcia closed #473
  • Jan 11 18:25

    mvidalgarcia on maint-0.7

    helm: expose user_confirmation … (compare)

  • Jan 11 15:57
    mvidalgarcia synchronize #473
  • Jan 11 09:08
    mvidalgarcia synchronize #473
  • Jan 11 08:59
    mvidalgarcia edited #473
  • Jan 11 08:59
    mvidalgarcia edited #473
  • Jan 08 15:52
    mvidalgarcia edited #473
  • Jan 08 15:49
    codecov-io commented #473
  • Jan 08 15:49
    mvidalgarcia edited #473
  • Jan 08 15:49
    mvidalgarcia edited #473
  • Jan 08 15:48
    mvidalgarcia edited #473
  • Jan 08 15:48
    mvidalgarcia opened #473
  • Jan 06 12:19
    diegodelemos closed #471
  • Jan 06 12:19

    diegodelemos on maint-0.7

    helm: control ui signup form vi… (compare)

  • Jan 04 12:48
    audrium synchronize #471
  • Jan 04 10:53
    audrium synchronize #471
  • Jan 04 10:47
    audrium synchronize #471
  • Jan 04 10:45
    codecov-io commented #471
  • Jan 04 10:43
    audrium synchronize #471
Alexander Myltsev
@alexander-myltsev
it’s (b). I’m looking for a working REANA cluster to get better understanding how it works from a user perspective. I’m also OK with publicly available REANA cluster if there’s any
running the cluster locally in a bunch of Docker containers is fine too
Alexander Myltsev
@alexander-myltsev
@tiborsimko :arrow_up: ?
Tibor Šimko
@tiborsimko
@alexander-myltsev Sorry, I have been busy. In your scenario, I would recommend to use Kind, it works well for local deployments and is swifter than Minikube. You can install Kind e.g. yay -S kind-bin on Arch Linux and you are ready to go. In order to have the very latest bleeding-edge REANA cluster, you can proceed in a CI style as follows: (note that it may take up to half an hour to build everything from scratch the first time)
# create Python virtual environment
mkvirtualenv reana
# clone reana sources
cdvirtualenv && mkdir src && cd src
git clone -b kind --depth 1 https://github.com/reanahub/reana
cd reana
# install reana-dev helper script
pip install .
# clone the other necessary REANA sources
reana-dev git-clone -u anonymous -c CLUSTER -c CLIENT -c r-d-helloworld
# deploy REANA cluster and run hello world example
reana-dev run-ci --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld
If this succeeds, you can run eval $(reana-dev client-setup-environment) and use reana-client run -w myanalysis on your analyses, as with Minikube. (note the slight syntax change: "reana-dev client-setup-environment")
Alexander Myltsev
@alexander-myltsev
OK, thanks. speaking of Kind, how long should I go through Quick Start? is it enough to kind create cluster to make reana work?
ah, reana-dev run-ci… is creating it, I see
Alexander Myltsev
@alexander-myltsev

@tiborsimko the error:

[2020-07-10T20:38:10] reana: kind load docker-image kozea/wdb:3.2.5
Image: "kozea/wdb:3.2.5" with ID "sha256:8e49b70c16baa5f079863ae94d732a6aa0ab5f103ef72d0db050b5a0e6c3de77" not yet present on node "kind-control-plane", loading...
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: unpacking docker.io/kozea/wdb:3.2.5 (sha256:eae1d26ddf8624a951e1444f462ef0b0a3ca078dda987257a40f9528691cb24f)...time="2020-07-10T17:39:00Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
time="2020-07-10T17:39:00Z" level=warning msg="extraction snapshot removal failed" error="write /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
ctr: failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown
[2020-07-10T20:38:10] reana: Command 'kind load docker-image kozea/wdb:3.2.5' returned non-zero exit status 1.
[2020-07-10T20:37:54] reana: Command 'reana-dev kind-load-docker-image -c reana' returned non-zero exit status 1.
[2020-07-10T20:34:05] reana: Command 'reana-dev cluster-create' returned non-zero exit status 1.

there is plenty of RAM and SSD space on my machine

Tibor Šimko
@tiborsimko
Are you on macOS? If yes, you'd have to go to Docker Desktop app and change Preferences -> Resources to allocate more RAM and more SSD to Docker, and restart Docker. I think at least 8 GB RAM and at least 60 GB space.

Another trick that I'm using is sharing /var/reana with host. I.e. you can create empty /var/reana on your host, and use:

reana-dev cluster-delete
reana-dev run-ci -m /var/reana:/var/reana --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld

In this way Kind will host-mount /var/reana to the cluster, so there will be less demands on Docker volume disk space, and you'll be able to access /var/reana/... workspaces comfortably from within your host.

Alexander Myltsev
@alexander-myltsev

@tiborsimko

I launched the cluster and passed tutorial. thank you!

I have enough SSD space to launch it without trick. but I’ll keep in mind it when required

how should I delete everything that was installed? is reana-dev cluster-delete enough?
Tibor Šimko
@tiborsimko
Yes, that is enough for the cluster. If you want to delete also the docker images on your host, you can do reana-dev docker-rmi.
Adam Morris
@abmorris

Hi. I'm trying to set up a mini REANA cluster on CERN Openstack following this tutorial: https://reana-cluster.readthedocs.io/en/latest/gettingstarted.html

Other than a few minor things, I managed to get right down to the last command before encountering an error:

$ reana-cluster -f reana-cluster-CERN.yaml init
Error: Couldn't load repositories file (/afs/cern.ch/user/a/admorris/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)
[ERROR] Traefik initialization failed 
 Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1.

I documented what I did, and the output, as I went along:
https://codimd.web.cern.ch/7jlGFHh1Sh6vKmG7NGsPTg/

oh wait I didn't spot the start of the error message
I'll try what it suggests
Adam Morris
@abmorris
I called helm init and waited a while. Now I have (without the traceback):
$ reana-cluster -f reana-cluster-CERN.yaml init
Error: This command needs 1 argument: chart name
[ERROR] Traefik initialization failed 
 Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1.
Marco Vidal García
@mvidalgarcia
@abmorris are you using latest master REANA components? reana-cluster is going to be deprecated in favor of helm deployment, you can follow instructions in our new (still WIP) docs site: https://docs.reana.io/development/deploying-locally/
Adam Morris
@abmorris
I guess I'm not
ok I will try a local deployment before moving to openstack
thanks
Adam Morris
@abmorris
Hi. So I've been added to the LHCb Analysis Preservation Openstack project and I see there's no Share/Share Storage quota. I requested some CephFS space, but I was asked by our computing team why I can't use EOS instead. I'm not sure I can answer definitively.
I'm also thinking whether it's possible to use a volume + NFS
6 replies
Adam Morris
@abmorris
Is there a helm 3.2.x installation on lxplus?
3 replies
Adam Morris
@abmorris

This is where I am with the NFS provisioner: https://codimd.web.cern.ch/s/ryXsju3gw#NFS-share

I feel like I'm missing something. When I describe pods, the one called db is in a crash loop, and both the NFS provisioner and REANA server pod have warnings about pod has unbound immediate PersistentVolumeClaims

Adam Morris
@abmorris
right, I had to create a PersistentVolume
Adam Morris
@abmorris

ok, I'm stuck. Could someone check that CodiMD link ^ and tell me where I've gone wrong?
I have an OpenStack volume connected to a PersistentVolume. The NFS provisioner is running and the PersistentVolumeClaim created by REANA is attached successfully.

The db pod is still in a crash loop, while the server and workflow-controller pods are stuck in ContainerCreating

6 replies
Tibor Šimko
@tiborsimko
@abmorris Two meta-comments while inspecting the PVC situation: (1) If you plan to test some LHCb workflows, would it be feasible to use reana.cern.ch instance? There is a possibility to access restricted resources via keytab if you wish, we have some ATLAS and CMS examples. We could perhaps talk in CERN's REANA Mattermost channel about your particular use case? (2) Concerning supporting EOS, it is currently necessary to have some Ceph/NFS shared storage for cluster nodes indeed. One can do stage-in/stage-out of data from/to EOS already, but this is not practical to do for all the workflow steps... We are planning to look into a possibility of running user workflows directly on the user EOS workspace; that might perhaps be of interest?
5 replies
Chris Hollowell
@hollowec
HI all. We've deployed a test REANA instance on a k8s cluster at BNL. Ideally, we would want to run this in our openshift cluster instead. Has anyone successfully done this without admin-level privileges in openshift? I'm getting some errors about access to clusterrolebindings when using helm to install (potentially related to https://github.com/reanahub/reana/pull/291/). We would need to use openshift routes instead of Traefik ingress also. Also does anyone have documentation/an example of configuring REANA to use K5/non-CERN IDP auth, instead of local accounts?
Stian Soiland-Reyes
@stain
Hi, is https://reana.cern.ch/ meant to be down/firewalled?
Tibor Šimko
@tiborsimko
Hi @stain, yes reana.cern.ch is still behind CERN firewall and used for physics workflows. We don't have any public "demo" site open to non-CERN users yet, but hopefully this will change in the coming weeks.
Tibor Šimko
@tiborsimko
Hi @hollowec, sorry for the late reply! It is now possible to install REANA easily into a particular namespace, so some of the things necessary for the RIVER deployment are now available out of the box. However we haven't looked systematically into the limited privileges yet... We can revive that thread though! Also, haven't tried with OpenShift yet (this could be less work) or with non-local non-CERN SSO options (this could be more work). Have tried to deploy on Google Cloud GKE, that was working well out of the box (using local accounts).
Chris Hollowell
@hollowec
OK thanks @tiborsimko. It would be nice to integrate with our IDP if possible, and that would likely be needed before this could be adopted on a larger scale at BNL. If we are using local accounts, is there a way to disable the sign-up functionality on the web interface? I would prefer to create users manually on the reana server command-line (flask user-create) as needed. Does user-create allow the assignment of a web-ui password? Also, is there a way for users to change their password via the web interface?
If you can provide an openshift-compatible helm chart, or associated helm values needed for use with openshift with the existing chart that would be great.
Tibor Šimko
@tiborsimko
We used to have it exactly as you described in 0.6 and past releases, that is there was no sign-up on the web and it was only the admin person who could create user accounts. As of 0.7 release, users can sign-up by themselves, but please note that this action is not sufficient to launch any workflows yet; the user has to request access token via a web interface button, which will send an email to the admin, and it up to the admin to grant or refuse the user requests. Only after the admin grants token can the signed-up user start using the system. Would it be sufficient for you like this? Or would you stlil like to block sign-up? If so, please open an issue on GitHub, we can make the sign-up feature optional upon deployment!
Chris Hollowell
@hollowec
Hi @tiborsimko , thanks yes I saw I needed to grant a user a token before they can use the system. The issue is if the web interface were to be opened to the world, someone could go through and create accounts for others, because there is no email verification needed to create the account. For instance, I could register an account for someone, say "someuser@bnl.gov", and then the real "someuser@bnl.gov" can't create an account unless I delete the original bogus user first. Also if it was fully open, external entities could fill up the user database with entries. Therefore, I think the signup should be an option: I will open a ticket in github. Thanks!
Mattias de Hollander
@mdehollander
Hi there. I have been keeping an eye on this project for a while and I really like it :) I also have been following the development on Renku. I like features from both. 1) Reana that you can submit cwl workflows to a cluster 2) With Renku you can launch a interactive Rstudio/Jupyter session. In the past there have been attempts (in 2018) to integrate both systems what I found is correct. There is also a integration label on github (https://github.com/reanahub/reana/issues?q=label%3Aintegration%2Frenku+). What is the current stage/plans for this integration. And now that Reana-UI has an option to launch interactive session for Jupyter (reanahub/reana-ui#77) would this integration still be needed? Thanks for the feedback, looking forward to start playing with it.
Kyle Cranmer
@cranmer
HI all... I wanted to discuss a REANA deployment at NYU using the Slurm backend. I'm aware of the efforts there, but I'm not sure what the status is. When I look at the docs I only see references to Slurm at CERN:
http://docs.reana.io/advanced-usage/compute-backends/slurm/
and I don't see anything about deploying at scale:
http://docs.reana.io/development/deploying-at-scale/
I also don't see much about deploying for HTCondor in the docs, and I think @hollowec would also be interested in that.
Tibor Šimko
@tiborsimko
@mdehollander Hi and sorry for the late reply! We've been in touch with RENKU in the past, exactly around the idea of running CWL workflows produced by RENKU on the REANA backend... but we haven't pursued the discussions lately. REANA focuses mostly on running the batch workflows, so the support for running interactive notebooks is a bit "secondary" there. The current aim is basically to allow people to open a notebook alongside running batch workflow for some quick explorations of intermediate data on the workflow's workspace. Kind of getting a remote shell on the workspace, if you will... That said, it might be interesting to bridge the notebook world and the workflow world better, e.g. to allow dispatching REANA workflows directly from the notebook, for those users who use notebooks as the main entry interface. We plan to enrich web interface with some R/W features in 2021, so we could perhaps revive this topic soon!
Tibor Šimko
@tiborsimko
@cranmer Hi, REANA contains an job controller abstraction notion that is described in this mini-paper: http://cds.cern.ch/record/2696223/files/CERN-IT-2019-004.pdf and This allows to dispatch jobs to CERN HTCondor and CERN Slurm installations (using Kerberos authentication). We publish corresponding Docker images for CERN HTC/HPC backends here: https://hub.docker.com/r/reanahub/reana-job-controller-htcondorcern-slurmcern/tags?page=1&ordering=last_updated The deployment is easy, enabling the backends in the configuration and replacing standard reanahub/reana-job-controller image (working only with Kubernetes) by the one that contains CERN HPC/HTC support. IOW, one would basically use the same Helm deployment recipe, only change the image. WRT adding support for the NYU cluster, it should be possible by sub-classing the abstract job controller class and overriding methods to use NYU specifics instead of CERN specifics, see Figure 2 in the cited PDF mini-paper. This was done successfully by the SCAILFIN@NotreDame team for the VC3, so I guess the abstraction of the design is tested to be relatively CERN-agnostic already. If you also use Kerberos kinit for NYU Slurm, it should not be that difficult an adaptation I guess... Perhaps just to replace the name of the head node etc... We could take this as an occasion to better parametrise Slurm backend
Kyle Cranmer
@cranmer
Thanks @tiborsimko
Chris Hollowell
@hollowec
Thanks @tiborsimko I will take a look at the document!
Tibor Šimko
@tiborsimko
@hollowec Please see also our latest paper from CHEP2019 that has some more details on this https://www.epj-conferences.org/articles/epjconf/abs/2020/21/epjconf_chep2020_06041/epjconf_chep2020_06041.html
Kyle Cranmer
@cranmer
Thanks again @tiborsimko that's also useful. Is there a plan to add this information on SLURM to the admin documentation ?
Tibor Šimko
@tiborsimko
Indeed, we have focused mostly on user-level documentation, the admin-level regarding HTC/HPC deployment is lacking behind...
Chris Hollowell
@hollowec
Thanks @tiborsimko I will take a look a the CHEP paper
Mattias de Hollander
@mdehollander

@mdehollander Hi and sorry for the late reply! We've been in touch with RENKU in the past, exactly around the idea of running CWL workflows produced by RENKU on the REANA backend... but we haven't pursued the discussions lately. REANA focuses mostly on running the batch workflows, so the support for running interactive notebooks is a bit "secondary" there. The current aim is basically to allow people to open a notebook alongside running batch workflow for some quick explorations of intermediate data on the workflow's workspace. Kind of getting a remote shell on the workspace, if you will... That said, it might be interesting to bridge the notebook world and the workflow world better, e.g. to allow dispatching REANA workflows directly from the notebook, for those users who use notebooks as the main entry interface. We plan to enrich web interface with some R/W features in 2021, so we could perhaps revive this topic soon!

Thanks for the update @tiborsimko