Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 02 14:16
    codecov-io commented #293
  • Dec 02 14:13
    hollowec synchronize #293
  • Dec 02 14:10
    hollowec opened #293
  • Dec 02 13:50
    hollowec opened #469
  • Nov 26 14:40
    audrium closed #439
  • Nov 26 14:40
    audrium commented #439
  • Nov 26 09:37
    diegodelemos labeled #468
  • Nov 26 09:37
    diegodelemos opened #468
  • Nov 26 09:18
    diegodelemos labeled #467
  • Nov 26 09:18
    diegodelemos opened #467
  • Nov 25 17:19
    diegodelemos closed #240
  • Nov 25 17:19
    diegodelemos commented #240
  • Nov 25 14:49

    mvidalgarcia on gh-pages

    index.yaml: 0.8.0-alpha.1 (compare)

  • Nov 25 14:26

    mvidalgarcia on 0.8.0-alpha.1

    (compare)

  • Nov 25 14:25
    mvidalgarcia closed #466
  • Nov 25 14:25

    mvidalgarcia on master

    release: 0.8.0a1 (compare)

  • Nov 25 14:19
    mvidalgarcia synchronize #466
  • Nov 25 14:18
    mvidalgarcia synchronize #466
  • Nov 25 14:05
    codecov-io commented #466
  • Nov 25 14:04
    mvidalgarcia synchronize #466
Umair Sarfraz
@laumair

Hi, I was looking at https://github.com/reanahub/reana-ui and was wondering if external contributions are accepted at this stage of the project?

I tried setting up the project but looks like the backend can’t be accessed outside the CERN network.

Tibor Šimko
@tiborsimko
Hi, the REANA project is still young but external contributions are welcome! What exactly did you have in mind for REANA UI? (P.S. Yes, the reana.cern.ch instance is currently only available inside the CERN network. One can install REANA on premises though, several organisations already did.)
Umair Sarfraz
@laumair
@tiborsimko I don’t have anything specific in mind at this point. Right now, I am just exploring and was aiming to help with some of the issues/tasks on the repo. @mvidalgarcia mentioned in DM that there is a way to setup the dev environment to develop locally.
Marco Vidal García
@mvidalgarcia
This is the recipe I'm using at the moment to work from home due to COVID-19 situation, what it does is basically bypassing the SSO auth by providing your access token on each request from the UI.
We should find a better solution eventually, but for now, this allows local development ignoring SSO.
Umair Sarfraz
@laumair
Thanks, I’ll check it out.
Alexander Myltsev
@alexander-myltsev

hi
I'm trying to launch reana cluster locally via minikube

I launched minikube start --feature-gates="TTLAfterFinished=true" -p reana-test-env

then I'm walking through https://github.com/reanahub/reana-demo-helloworld

and I'm stuck on

$ export REANA_SERVER_URL=https://reana.cern.ch/
$ export REANA_ACCESS_TOKEN=XXXXXXX

what are they for the local launch?

Tibor Šimko
@tiborsimko
@alexander-myltsev Have you installed reana-cluster or reana-dev helper tools? If yes, you can do things like eval $(reana-cluster setup-environment) which will set up the two environment variables for you. P.S. Depends on which REANA version you are trying locally, reana-cluster with 0.6.1, reana-dev with master branch. (reana-cluster has been deprecated when we switched to Helm.)
Alexander Myltsev
@alexander-myltsev
@tiborsimko I’m using reana-dev, but it fails with the error that I don’t understand:
source $(reana-dev setup-environment)
Error from server (NotFound): services "reana-test-env-traefik" not found
Environment variables could not be generated:
not enough values to unpack (expected 2, got 0)
source: not enough arguments
Tibor Šimko
@tiborsimko
Are you using latest master branch everywhere? Have you modified Helm charts in some way? You can check the status of your local branches via:
$ reana-dev git-fetch -c ALL
$ reana-dev git-status -c ALL -s
Alexander Myltsev
@alexander-myltsev
in the reana folder downloaded from github, I have an error:
$ reana-dev git-fetch -c ALL
[2020-07-10T10:11:15] reana-ui: git fetch upstream
Traceback (most recent call last):
  File "/reana-test/reana-test-env/bin/reana-dev", line 11, in <module>
    load_entry_point('reana==0.7.0.dev20200224', 'console_scripts', 'reana-dev')()
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/reana/cli.py", line 1174, in git_fetch
    run_command(cmd, component)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/reana/cli.py", line 662, in run_command
    os.chdir(get_srcdir(component))
FileNotFoundError: [Errno 2] No such file or directory: '/reana-test/reana-ui’
Tibor Šimko
@tiborsimko
OK, the above two reana-dev git-* commands would necessitate to have forked and cloned all the REANA repositories... which is perhaps not what you are looking for? There are two options: (a) are you looking for a full developer-oriented setup that would allow you to modify REANA cluster sources such as workflow controller or job controller? Or (b) are you looking at having latest REANA installed locally in order to develop analysis workflows and without necessarily looking into altering REANA cluster component sources? Please let me know which of the two, we can try different recipes
Alexander Myltsev
@alexander-myltsev
it’s (b). I’m looking for a working REANA cluster to get better understanding how it works from a user perspective. I’m also OK with publicly available REANA cluster if there’s any
running the cluster locally in a bunch of Docker containers is fine too
Alexander Myltsev
@alexander-myltsev
@tiborsimko :arrow_up: ?
Tibor Šimko
@tiborsimko
@alexander-myltsev Sorry, I have been busy. In your scenario, I would recommend to use Kind, it works well for local deployments and is swifter than Minikube. You can install Kind e.g. yay -S kind-bin on Arch Linux and you are ready to go. In order to have the very latest bleeding-edge REANA cluster, you can proceed in a CI style as follows: (note that it may take up to half an hour to build everything from scratch the first time)
# create Python virtual environment
mkvirtualenv reana
# clone reana sources
cdvirtualenv && mkdir src && cd src
git clone -b kind --depth 1 https://github.com/reanahub/reana
cd reana
# install reana-dev helper script
pip install .
# clone the other necessary REANA sources
reana-dev git-clone -u anonymous -c CLUSTER -c CLIENT -c r-d-helloworld
# deploy REANA cluster and run hello world example
reana-dev run-ci --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld
If this succeeds, you can run eval $(reana-dev client-setup-environment) and use reana-client run -w myanalysis on your analyses, as with Minikube. (note the slight syntax change: "reana-dev client-setup-environment")
Alexander Myltsev
@alexander-myltsev
OK, thanks. speaking of Kind, how long should I go through Quick Start? is it enough to kind create cluster to make reana work?
ah, reana-dev run-ci… is creating it, I see
Alexander Myltsev
@alexander-myltsev

@tiborsimko the error:

[2020-07-10T20:38:10] reana: kind load docker-image kozea/wdb:3.2.5
Image: "kozea/wdb:3.2.5" with ID "sha256:8e49b70c16baa5f079863ae94d732a6aa0ab5f103ef72d0db050b5a0e6c3de77" not yet present on node "kind-control-plane", loading...
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: unpacking docker.io/kozea/wdb:3.2.5 (sha256:eae1d26ddf8624a951e1444f462ef0b0a3ca078dda987257a40f9528691cb24f)...time="2020-07-10T17:39:00Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
time="2020-07-10T17:39:00Z" level=warning msg="extraction snapshot removal failed" error="write /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
ctr: failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown
[2020-07-10T20:38:10] reana: Command 'kind load docker-image kozea/wdb:3.2.5' returned non-zero exit status 1.
[2020-07-10T20:37:54] reana: Command 'reana-dev kind-load-docker-image -c reana' returned non-zero exit status 1.
[2020-07-10T20:34:05] reana: Command 'reana-dev cluster-create' returned non-zero exit status 1.

there is plenty of RAM and SSD space on my machine

Tibor Šimko
@tiborsimko
Are you on macOS? If yes, you'd have to go to Docker Desktop app and change Preferences -> Resources to allocate more RAM and more SSD to Docker, and restart Docker. I think at least 8 GB RAM and at least 60 GB space.

Another trick that I'm using is sharing /var/reana with host. I.e. you can create empty /var/reana on your host, and use:

reana-dev cluster-delete
reana-dev run-ci -m /var/reana:/var/reana --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld

In this way Kind will host-mount /var/reana to the cluster, so there will be less demands on Docker volume disk space, and you'll be able to access /var/reana/... workspaces comfortably from within your host.

Alexander Myltsev
@alexander-myltsev

@tiborsimko

I launched the cluster and passed tutorial. thank you!

I have enough SSD space to launch it without trick. but I’ll keep in mind it when required

how should I delete everything that was installed? is reana-dev cluster-delete enough?
Tibor Šimko
@tiborsimko
Yes, that is enough for the cluster. If you want to delete also the docker images on your host, you can do reana-dev docker-rmi.
Adam Morris
@abmorris

Hi. I'm trying to set up a mini REANA cluster on CERN Openstack following this tutorial: https://reana-cluster.readthedocs.io/en/latest/gettingstarted.html

Other than a few minor things, I managed to get right down to the last command before encountering an error:

$ reana-cluster -f reana-cluster-CERN.yaml init
Error: Couldn't load repositories file (/afs/cern.ch/user/a/admorris/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)
[ERROR] Traefik initialization failed 
 Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1.

I documented what I did, and the output, as I went along:
https://codimd.web.cern.ch/7jlGFHh1Sh6vKmG7NGsPTg/

oh wait I didn't spot the start of the error message
I'll try what it suggests
Adam Morris
@abmorris
I called helm init and waited a while. Now I have (without the traceback):
$ reana-cluster -f reana-cluster-CERN.yaml init
Error: This command needs 1 argument: chart name
[ERROR] Traefik initialization failed 
 Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1.
Marco Vidal García
@mvidalgarcia
@abmorris are you using latest master REANA components? reana-cluster is going to be deprecated in favor of helm deployment, you can follow instructions in our new (still WIP) docs site: https://docs.reana.io/development/deploying-locally/
Adam Morris
@abmorris
I guess I'm not
ok I will try a local deployment before moving to openstack
thanks
Adam Morris
@abmorris
Hi. So I've been added to the LHCb Analysis Preservation Openstack project and I see there's no Share/Share Storage quota. I requested some CephFS space, but I was asked by our computing team why I can't use EOS instead. I'm not sure I can answer definitively.
I'm also thinking whether it's possible to use a volume + NFS
6 replies
Adam Morris
@abmorris
Is there a helm 3.2.x installation on lxplus?
3 replies
Adam Morris
@abmorris

This is where I am with the NFS provisioner: https://codimd.web.cern.ch/s/ryXsju3gw#NFS-share

I feel like I'm missing something. When I describe pods, the one called db is in a crash loop, and both the NFS provisioner and REANA server pod have warnings about pod has unbound immediate PersistentVolumeClaims

Adam Morris
@abmorris
right, I had to create a PersistentVolume
Adam Morris
@abmorris

ok, I'm stuck. Could someone check that CodiMD link ^ and tell me where I've gone wrong?
I have an OpenStack volume connected to a PersistentVolume. The NFS provisioner is running and the PersistentVolumeClaim created by REANA is attached successfully.

The db pod is still in a crash loop, while the server and workflow-controller pods are stuck in ContainerCreating

6 replies
Tibor Šimko
@tiborsimko
@abmorris Two meta-comments while inspecting the PVC situation: (1) If you plan to test some LHCb workflows, would it be feasible to use reana.cern.ch instance? There is a possibility to access restricted resources via keytab if you wish, we have some ATLAS and CMS examples. We could perhaps talk in CERN's REANA Mattermost channel about your particular use case? (2) Concerning supporting EOS, it is currently necessary to have some Ceph/NFS shared storage for cluster nodes indeed. One can do stage-in/stage-out of data from/to EOS already, but this is not practical to do for all the workflow steps... We are planning to look into a possibility of running user workflows directly on the user EOS workspace; that might perhaps be of interest?
5 replies
Chris Hollowell
@hollowec
HI all. We've deployed a test REANA instance on a k8s cluster at BNL. Ideally, we would want to run this in our openshift cluster instead. Has anyone successfully done this without admin-level privileges in openshift? I'm getting some errors about access to clusterrolebindings when using helm to install (potentially related to https://github.com/reanahub/reana/pull/291/). We would need to use openshift routes instead of Traefik ingress also. Also does anyone have documentation/an example of configuring REANA to use K5/non-CERN IDP auth, instead of local accounts?
Stian Soiland-Reyes
@stain
Hi, is https://reana.cern.ch/ meant to be down/firewalled?
Tibor Šimko
@tiborsimko
Hi @stain, yes reana.cern.ch is still behind CERN firewall and used for physics workflows. We don't have any public "demo" site open to non-CERN users yet, but hopefully this will change in the coming weeks.
Tibor Šimko
@tiborsimko
Hi @hollowec, sorry for the late reply! It is now possible to install REANA easily into a particular namespace, so some of the things necessary for the RIVER deployment are now available out of the box. However we haven't looked systematically into the limited privileges yet... We can revive that thread though! Also, haven't tried with OpenShift yet (this could be less work) or with non-local non-CERN SSO options (this could be more work). Have tried to deploy on Google Cloud GKE, that was working well out of the box (using local accounts).
Chris Hollowell
@hollowec
OK thanks @tiborsimko. It would be nice to integrate with our IDP if possible, and that would likely be needed before this could be adopted on a larger scale at BNL. If we are using local accounts, is there a way to disable the sign-up functionality on the web interface? I would prefer to create users manually on the reana server command-line (flask user-create) as needed. Does user-create allow the assignment of a web-ui password? Also, is there a way for users to change their password via the web interface?
If you can provide an openshift-compatible helm chart, or associated helm values needed for use with openshift with the existing chart that would be great.
Tibor Šimko
@tiborsimko
We used to have it exactly as you described in 0.6 and past releases, that is there was no sign-up on the web and it was only the admin person who could create user accounts. As of 0.7 release, users can sign-up by themselves, but please note that this action is not sufficient to launch any workflows yet; the user has to request access token via a web interface button, which will send an email to the admin, and it up to the admin to grant or refuse the user requests. Only after the admin grants token can the signed-up user start using the system. Would it be sufficient for you like this? Or would you stlil like to block sign-up? If so, please open an issue on GitHub, we can make the sign-up feature optional upon deployment!
Chris Hollowell
@hollowec
Hi @tiborsimko , thanks yes I saw I needed to grant a user a token before they can use the system. The issue is if the web interface were to be opened to the world, someone could go through and create accounts for others, because there is no email verification needed to create the account. For instance, I could register an account for someone, say "someuser@bnl.gov", and then the real "someuser@bnl.gov" can't create an account unless I delete the original bogus user first. Also if it was fully open, external entities could fill up the user database with entries. Therefore, I think the signup should be an option: I will open a ticket in github. Thanks!
Mattias de Hollander
@mdehollander
Hi there. I have been keeping an eye on this project for a while and I really like it :) I also have been following the development on Renku. I like features from both. 1) Reana that you can submit cwl workflows to a cluster 2) With Renku you can launch a interactive Rstudio/Jupyter session. In the past there have been attempts (in 2018) to integrate both systems what I found is correct. There is also a integration label on github (https://github.com/reanahub/reana/issues?q=label%3Aintegration%2Frenku+). What is the current stage/plans for this integration. And now that Reana-UI has an option to launch interactive session for Jupyter (reanahub/reana-ui#77) would this integration still be needed? Thanks for the feedback, looking forward to start playing with it.