by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 16:28

    tiborsimko on 0.7.0-alpha.1

    (compare)

  • 16:28

    reana-team on gh-pages

    Update index.yaml Signed-off-b… (compare)

  • 16:28

    reana-team on reana-0.7.0-alpha.1

    (compare)

  • 16:28
    tiborsimko closed #384
  • 16:28

    tiborsimko on master

    release: 0.7.0-alpha.1 (compare)

  • 16:23
    tiborsimko synchronize #384
  • 16:22
    tiborsimko closed #383
  • 16:22
    tiborsimko closed #382
  • 16:22

    tiborsimko on master

    installation: packaging depende… (compare)

  • 16:19
    tiborsimko synchronize #383
  • 16:18
    tiborsimko closed #380
  • 16:18
    tiborsimko closed #370
  • 16:18

    tiborsimko on master

    cli: release-docker * Closes #… (compare)

  • 16:15
    tiborsimko synchronize #380
  • 15:51

    tiborsimko on 0.7.0a1

    (compare)

  • 15:50
    tiborsimko closed #160
  • 15:50

    tiborsimko on master

    installation: bump shared modul… pip-compile upgrade of dependen… release: 0.7.0a1 (compare)

  • 15:47
    diegodelemos synchronize #380
  • 15:47

    tiborsimko on 0.7.0a1

    (compare)

  • 15:47
    tiborsimko closed #270
Tibor Šimko
@tiborsimko
As you for yesterday problems, here is general tip:
  • to debug client-side toubles, you can increase logging level option via -l argument to reana-client, for example reana-client -l DEBUG ping
  • to debug server-side troubles, see kubect logs for interesting pods, for example kubectl get pods and kubectl logs server-775d8f69f4-r6jzm which will display any errors and Python tracebacks
Lincoln Bryant
@LincolnBryant
thanks Tibor!
Lincoln Bryant
@LincolnBryant
@tiborsimko I made a PR to that effect here: reanahub/reana-cluster#261 . that command worked for me. (it does appear -e is needed, according to --help)
Diego
@diegodelemos
Hello @LincolnBryant! thanks a lot for spotting these outdated docs, since we are in the process of re-organising and fixing the REANA documentation, this change will be integrated into our new repo for docs as we do not use the REANA-Cluster package anymore (reanahub/reana#234). The new documentation will take shape by the end of next week.
Lincoln Bryant
@LincolnBryant
@diegodelemos that's great to hear- I was going to ask if you guys would be interested in having a full helm chart for REANA but it looks like you've already done that! We would love to be able to include it in our catalog for SLATE (https://slateci.io/) (which wraps around K8S and Helm). I haven't looked over your chart just yet but would you be open to contributions for a few additional config knobs? Our clusters are a bit different from the ones at CERN for example but fairly consistent among themselves
Tibor Šimko
@tiborsimko
Yes, our helm chart is a bit CERN-centric for now, but we are definitely interested in making it more customisable for various deployment scenarios in the future
Tibor Šimko
@tiborsimko
We are hiring! Join us in building @reanahub reproducible analysis platform! A new @CERN fellowship position for research software engineers and full-stack developers #Python #React #Kubernetes #SciWorkflows https://twitter.com/reanahub/status/1237311364332490760 https://jobs.smartrecruiters.com/CERN/743999709159905-computing-fellowship-research-software-engineer-reproducible-scientific-workflow-platform
Umair Sarfraz
@laumair

Hi, I was looking at https://github.com/reanahub/reana-ui and was wondering if external contributions are accepted at this stage of the project?

I tried setting up the project but looks like the backend can’t be accessed outside the CERN network.

Tibor Šimko
@tiborsimko
Hi, the REANA project is still young but external contributions are welcome! What exactly did you have in mind for REANA UI? (P.S. Yes, the reana.cern.ch instance is currently only available inside the CERN network. One can install REANA on premises though, several organisations already did.)
Umair Sarfraz
@laumair
@tiborsimko I don’t have anything specific in mind at this point. Right now, I am just exploring and was aiming to help with some of the issues/tasks on the repo. @mvidalgarcia mentioned in DM that there is a way to setup the dev environment to develop locally.
Marco Vidal García
@mvidalgarcia
This is the recipe I'm using at the moment to work from home due to COVID-19 situation, what it does is basically bypassing the SSO auth by providing your access token on each request from the UI.
We should find a better solution eventually, but for now, this allows local development ignoring SSO.
Umair Sarfraz
@laumair
Thanks, I’ll check it out.
Alexander Myltsev
@alexander-myltsev

hi
I'm trying to launch reana cluster locally via minikube

I launched minikube start --feature-gates="TTLAfterFinished=true" -p reana-test-env

then I'm walking through https://github.com/reanahub/reana-demo-helloworld

and I'm stuck on

$ export REANA_SERVER_URL=https://reana.cern.ch/
$ export REANA_ACCESS_TOKEN=XXXXXXX

what are they for the local launch?

Tibor Šimko
@tiborsimko
@alexander-myltsev Have you installed reana-cluster or reana-dev helper tools? If yes, you can do things like eval $(reana-cluster setup-environment) which will set up the two environment variables for you. P.S. Depends on which REANA version you are trying locally, reana-cluster with 0.6.1, reana-dev with master branch. (reana-cluster has been deprecated when we switched to Helm.)
Alexander Myltsev
@alexander-myltsev
@tiborsimko I’m using reana-dev, but it fails with the error that I don’t understand:
source $(reana-dev setup-environment)
Error from server (NotFound): services "reana-test-env-traefik" not found
Environment variables could not be generated:
not enough values to unpack (expected 2, got 0)
source: not enough arguments
Tibor Šimko
@tiborsimko
Are you using latest master branch everywhere? Have you modified Helm charts in some way? You can check the status of your local branches via:
$ reana-dev git-fetch -c ALL
$ reana-dev git-status -c ALL -s
Alexander Myltsev
@alexander-myltsev
in the reana folder downloaded from github, I have an error:
$ reana-dev git-fetch -c ALL
[2020-07-10T10:11:15] reana-ui: git fetch upstream
Traceback (most recent call last):
  File "/reana-test/reana-test-env/bin/reana-dev", line 11, in <module>
    load_entry_point('reana==0.7.0.dev20200224', 'console_scripts', 'reana-dev')()
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/reana/cli.py", line 1174, in git_fetch
    run_command(cmd, component)
  File "/reana-test/reana-test-env/lib/python3.7/site-packages/reana/cli.py", line 662, in run_command
    os.chdir(get_srcdir(component))
FileNotFoundError: [Errno 2] No such file or directory: '/reana-test/reana-ui’
Tibor Šimko
@tiborsimko
OK, the above two reana-dev git-* commands would necessitate to have forked and cloned all the REANA repositories... which is perhaps not what you are looking for? There are two options: (a) are you looking for a full developer-oriented setup that would allow you to modify REANA cluster sources such as workflow controller or job controller? Or (b) are you looking at having latest REANA installed locally in order to develop analysis workflows and without necessarily looking into altering REANA cluster component sources? Please let me know which of the two, we can try different recipes
Alexander Myltsev
@alexander-myltsev
it’s (b). I’m looking for a working REANA cluster to get better understanding how it works from a user perspective. I’m also OK with publicly available REANA cluster if there’s any
running the cluster locally in a bunch of Docker containers is fine too
Alexander Myltsev
@alexander-myltsev
@tiborsimko :arrow_up: ?
Tibor Šimko
@tiborsimko
@alexander-myltsev Sorry, I have been busy. In your scenario, I would recommend to use Kind, it works well for local deployments and is swifter than Minikube. You can install Kind e.g. yay -S kind-bin on Arch Linux and you are ready to go. In order to have the very latest bleeding-edge REANA cluster, you can proceed in a CI style as follows: (note that it may take up to half an hour to build everything from scratch the first time)
# create Python virtual environment
mkvirtualenv reana
# clone reana sources
cdvirtualenv && mkdir src && cd src
git clone -b kind --depth 1 https://github.com/reanahub/reana
cd reana
# install reana-dev helper script
pip install .
# clone the other necessary REANA sources
reana-dev git-clone -u anonymous -c CLUSTER -c CLIENT -c r-d-helloworld
# deploy REANA cluster and run hello world example
reana-dev run-ci --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld
If this succeeds, you can run eval $(reana-dev client-setup-environment) and use reana-client run -w myanalysis on your analyses, as with Minikube. (note the slight syntax change: "reana-dev client-setup-environment")
Alexander Myltsev
@alexander-myltsev
OK, thanks. speaking of Kind, how long should I go through Quick Start? is it enough to kind create cluster to make reana work?
ah, reana-dev run-ci… is creating it, I see
Alexander Myltsev
@alexander-myltsev

@tiborsimko the error:

[2020-07-10T20:38:10] reana: kind load docker-image kozea/wdb:3.2.5
Image: "kozea/wdb:3.2.5" with ID "sha256:8e49b70c16baa5f079863ae94d732a6aa0ab5f103ef72d0db050b5a0e6c3de77" not yet present on node "kind-control-plane", loading...
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: unpacking docker.io/kozea/wdb:3.2.5 (sha256:eae1d26ddf8624a951e1444f462ef0b0a3ca078dda987257a40f9528691cb24f)...time="2020-07-10T17:39:00Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
time="2020-07-10T17:39:00Z" level=warning msg="extraction snapshot removal failed" error="write /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-102132277-rjAc sha256:2d9c829ae3f7ff3e148e5c7c3a1cf378b0f90b79035e2fe9a8d78c63ccde4c89"
ctr: failed to extract layer sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs/usr/lib/python2.7/collections.py: no space left on device: unknown
[2020-07-10T20:38:10] reana: Command 'kind load docker-image kozea/wdb:3.2.5' returned non-zero exit status 1.
[2020-07-10T20:37:54] reana: Command 'reana-dev kind-load-docker-image -c reana' returned non-zero exit status 1.
[2020-07-10T20:34:05] reana: Command 'reana-dev cluster-create' returned non-zero exit status 1.

there is plenty of RAM and SSD space on my machine

Tibor Šimko
@tiborsimko
Are you on macOS? If yes, you'd have to go to Docker Desktop app and change Preferences -> Resources to allocate more RAM and more SSD to Docker, and restart Docker. I think at least 8 GB RAM and at least 60 GB space.

Another trick that I'm using is sharing /var/reana with host. I.e. you can create empty /var/reana on your host, and use:

reana-dev cluster-delete
reana-dev run-ci -m /var/reana:/var/reana --exclude-components=r-ui,r-a-vomsproxy -c r-d-helloworld

In this way Kind will host-mount /var/reana to the cluster, so there will be less demands on Docker volume disk space, and you'll be able to access /var/reana/... workspaces comfortably from within your host.

Alexander Myltsev
@alexander-myltsev

@tiborsimko

I launched the cluster and passed tutorial. thank you!

I have enough SSD space to launch it without trick. but I’ll keep in mind it when required

how should I delete everything that was installed? is reana-dev cluster-delete enough?
Tibor Šimko
@tiborsimko
Yes, that is enough for the cluster. If you want to delete also the docker images on your host, you can do reana-dev docker-rmi.
Adam Morris
@abmorris

Hi. I'm trying to set up a mini REANA cluster on CERN Openstack following this tutorial: https://reana-cluster.readthedocs.io/en/latest/gettingstarted.html

Other than a few minor things, I managed to get right down to the last command before encountering an error:

$ reana-cluster -f reana-cluster-CERN.yaml init
Error: Couldn't load repositories file (/afs/cern.ch/user/a/admorris/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)
[ERROR] Traefik initialization failed 
 Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/']' returned non-zero exit status 1.

I documented what I did, and the output, as I went along:
https://codimd.web.cern.ch/7jlGFHh1Sh6vKmG7NGsPTg/

oh wait I didn't spot the start of the error message
I'll try what it suggests
Adam Morris
@abmorris
I called helm init and waited a while. Now I have (without the traceback):
$ reana-cluster -f reana-cluster-CERN.yaml init
Error: This command needs 1 argument: chart name
[ERROR] Traefik initialization failed 
 Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1..
[ERROR] Command '['helm', 'install', 'reana-traefik', 'stable/traefik', '--namespace', 'kube-system', '--values', '/afs/cern.ch/work/a/admorris/reana/reana/lib/python3.6/site-packages/reana_cluster/configurations/helm/traefik/minikube.yaml']' returned non-zero exit status 1.
Marco Vidal García
@mvidalgarcia
@abmorris are you using latest master REANA components? reana-cluster is going to be deprecated in favor of helm deployment, you can follow instructions in our new (still WIP) docs site: https://docs.reana.io/development/deploying-locally/
Adam Morris
@abmorris
I guess I'm not
ok I will try a local deployment before moving to openstack
thanks
Adam Morris
@abmorris
Hi. So I've been added to the LHCb Analysis Preservation Openstack project and I see there's no Share/Share Storage quota. I requested some CephFS space, but I was asked by our computing team why I can't use EOS instead. I'm not sure I can answer definitively.
I'm also thinking whether it's possible to use a volume + NFS
6 replies
Adam Morris
@abmorris
Is there a helm 3.2.x installation on lxplus?
3 replies
Adam Morris
@abmorris

This is where I am with the NFS provisioner: https://codimd.web.cern.ch/s/ryXsju3gw#NFS-share

I feel like I'm missing something. When I describe pods, the one called db is in a crash loop, and both the NFS provisioner and REANA server pod have warnings about pod has unbound immediate PersistentVolumeClaims

Adam Morris
@abmorris
right, I had to create a PersistentVolume
Adam Morris
@abmorris

ok, I'm stuck. Could someone check that CodiMD link ^ and tell me where I've gone wrong?
I have an OpenStack volume connected to a PersistentVolume. The NFS provisioner is running and the PersistentVolumeClaim created by REANA is attached successfully.

The db pod is still in a crash loop, while the server and workflow-controller pods are stuck in ContainerCreating

6 replies
Tibor Šimko
@tiborsimko
@abmorris Two meta-comments while inspecting the PVC situation: (1) If you plan to test some LHCb workflows, would it be feasible to use reana.cern.ch instance? There is a possibility to access restricted resources via keytab if you wish, we have some ATLAS and CMS examples. We could perhaps talk in CERN's REANA Mattermost channel about your particular use case? (2) Concerning supporting EOS, it is currently necessary to have some Ceph/NFS shared storage for cluster nodes indeed. One can do stage-in/stage-out of data from/to EOS already, but this is not practical to do for all the workflow steps... We are planning to look into a possibility of running user workflows directly on the user EOS workspace; that might perhaps be of interest?
5 replies