by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 15:22
    tiborsimko labeled #263
  • 15:21
    tiborsimko labeled #263
  • 15:21
    tiborsimko opened #263
  • 15:16
    tiborsimko closed #260
  • 15:16
    tiborsimko closed #256
  • 15:16

    tiborsimko on master

    htcondor: migrate to myschedd (compare)

  • 14:53
    alintulu synchronize #260
  • 14:48
    alintulu synchronize #260
  • 14:45
    alintulu synchronize #260
  • 13:39
    tiborsimko closed #345
  • 13:39

    tiborsimko on kind

    helm: use maildev 1.1.0 docker … cli: used versioned demo exampl… installation: addition of PyYAM… and 5 more (compare)

  • 12:42
    alintulu synchronize #155
  • 12:32
    alintulu synchronize #155
  • 12:28
    alintulu synchronize #260
  • 12:24
    alintulu synchronize #260
  • 12:09
    tiborsimko assigned #262
  • 12:09
    tiborsimko opened #262
  • 12:07
    tiborsimko opened #156
  • 12:07
    tiborsimko assigned #156
  • 12:05
    tiborsimko synchronize #318
Tibor Šimko
@tiborsimko
  • We don't have any publicly accessible REANA cluster that you could use, although we were musing about setting one up; it would require some CPU accounting limits etc which we did not get to doing yet.
  • The web interface is something that we are actively developing these months, so while there is reana-ui component where you could see some mock-ups, it is better to wait for a more complete "live" version coming in February. We have user logins connected to GitLab for easy Continuous Integration use cases, we are working on workflow list and workflow run details visualisation. The important design principle is doing heavy stuff as React components, so that the UI could be eventually plugged into JupyterLab notebooks, so that one could easily launch workflows from within notebooks and monitor progress, for example.
Lukas
@lukasheinrich
hi all, just checking in again (sorry for being annoying :) ). I should be online until the end of the week, so if testing is possible at any point, I'k happy to do it
Tibor Šimko
@tiborsimko
@lukasheinrich We had a last-minute surprise with cvmfs/influxdb/fluentd killing the cluster (INC2249437), but as far as nobody would ask for CVMFS runtime resources, the cluster is ready for testing; more via email
Lukas
@lukasheinrich
awesome! I don't think I need cvmfs for now
Michael R. Crusoe
@mr-c
Hey all, congratulations on the latest release. The hybrid scheduling feature looks really cool! Are there plans to support hybrid scheduling with CWL?
Tibor Šimko
@tiborsimko
Hi @mr-c yes, the hybrid scheduling is fully supported with CWL already, via REANA-specific compute_backend hint (using kubernetes by default)
Michael R. Crusoe
@mr-c
@tiborsimko great news! I guess I misread the release notes
Francois Lanusse
@EiffL
Hi everyone, just saw the release notes of the latest version and couldn't be more excited to see the support for SLURM backend. I'm part of the LSST Dark Energy Science Collaboration ( https://lsstdesc.org ) and would absolutely love to be able to deploy our analysis pipelines using REANA!
Our main computing facility is the NERSC Cori machine at Berkeley National Lab (which runs SLURM for compute nodes, shifter for containers, and a separate k8 service called Spin), has anyone already successfully setup REANA there? And if not, any recommendations for how to go about it?
Tibor Šimko
@tiborsimko
@EiffL Sounds interesting! You may want to get in touch with Kenyi Hurtado and/or Cody Kankel from Notre Dame University who are actually developing REANA workflows for NESRC/Shifter (via VC3 integration). See some details in Kenyi's CHEP 2019 presentation here
Francois Lanusse
@EiffL
Fantastic! will contact them, thanks Tibor.
Lukas
@lukasheinrich
hi all.. for the workshop I'm trying out to run some workflows
is it a good time now or should I try later.. I seem to hit some issues with reana.cern.ch
Tibor Šimko
@tiborsimko
Yeah, now is good. If you hit some issues just ping us via MM
Lincoln Bryant
@LincolnBryant
hi all. is the TTLAfterFinished feature in Kubernetes strictly required for REANA?
Tibor Šimko
@tiborsimko
Hi @LincolnBryant , it is not strictly speaking required, but without this option when the user workflows would finish, the workflow engine and job pods wouldn't be automatically terminated, so the cluster admins would have to delete finished pods manually
Lincoln Bryant
@LincolnBryant
got it, thanks. That's OK for us for now, I think.
Lincoln Bryant
@LincolnBryant
I've got my server up now but things are seemingly broken..
I've got a REANA_SERVER_URL and REANA_ACCESS_TOKEN defiend, but when I run the example here: https://github.com/reanahub/reana-demo-helloworld#running-the-example-on-reana-cloud

(myreana) [16:52]:~/myreana $ reana-client create -w myanalysis
Usage: reana-client create [OPTIONS]
Try "reana-client create --help" for help.

Error: Invalid value for "-f" / "--file": Path "./reana.yaml" does not exist.

oh i tried -n as well
Tibor Šimko
@tiborsimko
It looks like you were in the myreana directory when running the example, so the reana.yaml was not found. You have to be located in the directory where the example (e.g. reana-demo-worldpopulation) when issuing reana-client commands.
Lincoln Bryant
@LincolnBryant
Hi Tibor, thanks. Here's where I get stuck now:

(myreana) [10:10]:~/myreana/reana-demo-helloworld $ ls
code data environments LICENSE README.rst reana-cwl-htcondorcern.yaml reana-cwl.yaml reana-htcondorcern.yaml reana-yadage.yaml reana.yaml workflow

(myreana) [10:10]:~/myreana/reana-demo-helloworld $ reana-client create -n myanalysis

Workflow could not be created:

Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)

Tibor Šimko
@tiborsimko
Do you use REANA stable release 0.6.0 or REANA development version from master branches on GitHub?
Lincoln Bryant
@LincolnBryant
(myreana) [10:13]:~/myreana/reana-demo-helloworld $ pip list | grep reana
reana-client 0.6.0
reana-commons 0.6.0
this is on a Fedora 30 machine with Python 3.7.5 in my virtualenv
Lincoln Bryant
@LincolnBryant
For the server I just installed with pip as well I believe, but I did hack on things to replace Traefik with nginx as the ingress.
but reana ping and all of that worked fine
Lincoln Bryant
@LincolnBryant
so I hacked on it a bit more, updated my Kubernetes cluster to v1.16.7, and put Traefik back and I'm now able to run a "reana-client create -n myanalysis"
Lincoln Bryant
@LincolnBryant
moving onto the next issue, I'm trying to create some users

from the docs: https://reana-cluster.readthedocs.io/en/latest/gettingstarted.html
so I run that:
[root@river-c001 reana]# kubectl exec -ti $(kubectl get pods -l=app=server -o jsonpath='{.items[0].metadata.name}') -- flask users create -e lincolnb@uchicago.edu --admin-access-token $REANA_ACCESS_TOKEN
Defaulting container name to server.
Use 'kubectl describe pod/server-8c55d4649-j25f8 -n default' to see all of the containers in this pod.
Usage: flask users create [OPTIONS] EMAIL
Try "flask users create --help" for help.

Error: no such option: -e
command terminated with exit code 2

indeed running help gives me:
[root@river-c001 reana]# kubectl exec -ti $(kubectl get pods -l=app=server -o jsonpath='{.items[0].metadata.name}') -- flask users create --help
Defaulting container name to server.
Use 'kubectl describe pod/server-8c55d4649-j25f8 -n default' to see all of the containers in this pod.
Usage: flask users create [OPTIONS] EMAIL

Create a user.

Options:
--password TEXT
-a, --active
--help Show this message and exit.

This time the only thing I've changed in my REANA deployment is that my Traefik installation uses LoadBalancer instead of NodePort, everything else is completely vanilla. reana-cluster ping and reana-client create -n blah work just fine
Tibor Šimko
@tiborsimko
The help page seems outdated, there is no more flask create -e johndoes@example.org, we have added SSO login (via Invenio digital library framework) and now the syntax is different, please see the example at https://github.com/reanahub/reana-server/blob/master/scripts/setup
(i.e. do not use -e, just supply email address directly)
As you for yesterday problems, here is general tip:
  • to debug client-side toubles, you can increase logging level option via -l argument to reana-client, for example reana-client -l DEBUG ping
Tibor Šimko
@tiborsimko
  • to debug server-side troubles, see kubect logs for interesting pods, for example kubectl get pods and kubectl logs server-775d8f69f4-r6jzm which will display any errors and Python tracebacks
Lincoln Bryant
@LincolnBryant
thanks Tibor!
Lincoln Bryant
@LincolnBryant
@tiborsimko I made a PR to that effect here: reanahub/reana-cluster#261 . that command worked for me. (it does appear -e is needed, according to --help)
Diego
@diegodelemos
Hello @LincolnBryant! thanks a lot for spotting these outdated docs, since we are in the process of re-organising and fixing the REANA documentation, this change will be integrated into our new repo for docs as we do not use the REANA-Cluster package anymore (reanahub/reana#234). The new documentation will take shape by the end of next week.
Lincoln Bryant
@LincolnBryant
@diegodelemos that's great to hear- I was going to ask if you guys would be interested in having a full helm chart for REANA but it looks like you've already done that! We would love to be able to include it in our catalog for SLATE (https://slateci.io/) (which wraps around K8S and Helm). I haven't looked over your chart just yet but would you be open to contributions for a few additional config knobs? Our clusters are a bit different from the ones at CERN for example but fairly consistent among themselves
Tibor Šimko
@tiborsimko
Yes, our helm chart is a bit CERN-centric for now, but we are definitely interested in making it more customisable for various deployment scenarios in the future
Tibor Šimko
@tiborsimko
We are hiring! Join us in building @reanahub reproducible analysis platform! A new @CERN fellowship position for research software engineers and full-stack developers #Python #React #Kubernetes #SciWorkflows https://twitter.com/reanahub/status/1237311364332490760 https://jobs.smartrecruiters.com/CERN/743999709159905-computing-fellowship-research-software-engineer-reproducible-scientific-workflow-platform
Umair Sarfraz
@laumair

Hi, I was looking at https://github.com/reanahub/reana-ui and was wondering if external contributions are accepted at this stage of the project?

I tried setting up the project but looks like the backend can’t be accessed outside the CERN network.

Tibor Šimko
@tiborsimko
Hi, the REANA project is still young but external contributions are welcome! What exactly did you have in mind for REANA UI? (P.S. Yes, the reana.cern.ch instance is currently only available inside the CERN network. One can install REANA on premises though, several organisations already did.)
Umair Sarfraz
@laumair
@tiborsimko I don’t have anything specific in mind at this point. Right now, I am just exploring and was aiming to help with some of the issues/tasks on the repo. @mvidalgarcia mentioned in DM that there is a way to setup the dev environment to develop locally.
Marco Vidal García
@mvidalgarcia
This is the recipe I'm using at the moment to work from home due to COVID-19 situation, what it does is basically bypassing the SSO auth by providing your access token on each request from the UI.
We should find a better solution eventually, but for now, this allows local development ignoring SSO.
Umair Sarfraz
@laumair
Thanks, I’ll check it out.