6 replies
This is where I am with the NFS provisioner: https://codimd.web.cern.ch/s/ryXsju3gw#NFS-share
I feel like I'm missing something. When I describe pods, the one called db is in a crash loop, and both the NFS provisioner and REANA server pod have warnings about pod has unbound immediate PersistentVolumeClaims
ok, I'm stuck. Could someone check that CodiMD link ^ and tell me where I've gone wrong?
I have an OpenStack volume connected to a PersistentVolume. The NFS provisioner is running and the PersistentVolumeClaim created by REANA is attached successfully.
The db pod is still in a crash loop, while the server and workflow-controller pods are stuck in ContainerCreating
6 replies
reana.cern.ch instance? There is a possibility to access restricted resources via keytab if you wish, we have some ATLAS and CMS examples. We could perhaps talk in CERN's REANA Mattermost channel about your particular use case? (2) Concerning supporting EOS, it is currently necessary to have some Ceph/NFS shared storage for cluster nodes indeed. One can do stage-in/stage-out of data from/to EOS already, but this is not practical to do for all the workflow steps... We are planning to look into a possibility of running user workflows directly on the user EOS workspace; that might perhaps be of interest?
5 replies
flask user-create) as needed. Does user-create allow the assignment of a web-ui password? Also, is there a way for users to change their password via the web interface?
http://docs.reana.io/advanced-usage/compute-backends/slurm/
http://docs.reana.io/development/deploying-at-scale/
reanahub/reana-job-controller image (working only with Kubernetes) by the one that contains CERN HPC/HTC support. IOW, one would basically use the same Helm deployment recipe, only change the image. WRT adding support for the NYU cluster, it should be possible by sub-classing the abstract job controller class and overriding methods to use NYU specifics instead of CERN specifics, see Figure 2 in the cited PDF mini-paper. This was done successfully by the SCAILFIN@NotreDame team for the VC3, so I guess the abstraction of the design is tested to be relatively CERN-agnostic already. If you also use Kerberos kinit for NYU Slurm, it should not be that difficult an adaptation I guess... Perhaps just to replace the name of the head node etc... We could take this as an occasion to better parametrise Slurm backend
@mdehollander Hi and sorry for the late reply! We've been in touch with RENKU in the past, exactly around the idea of running CWL workflows produced by RENKU on the REANA backend... but we haven't pursued the discussions lately. REANA focuses mostly on running the batch workflows, so the support for running interactive notebooks is a bit "secondary" there. The current aim is basically to allow people to open a notebook alongside running batch workflow for some quick explorations of intermediate data on the workflow's workspace. Kind of getting a remote shell on the workspace, if you will... That said, it might be interesting to bridge the notebook world and the workflow world better, e.g. to allow dispatching REANA workflows directly from the notebook, for those users who use notebooks as the main entry interface. We plan to enrich web interface with some R/W features in 2021, so we could perhaps revive this topic soon!
Thanks for the update @tiborsimko
is there a way to disable the sign-up functionality on the web interface?
@hollowec We have just released REANA 0.7.2 which has a deployment option to disable the sign-up functionality. Please see https://blog.reana.io/posts/2021/reana-0.7.2/
Long time since the last post (2019!!!)... We are testing REANA at PIC, here in Barcelona. We have a version working with Kubernetes and nou our idea is to integrate REANA with our HTCondor system. We checked the documentation and the code and we saw that the default configuration provided is quite CERN dependant, and we are wondering if there is any detailed procedure to change the configuration to configure other Compute Backends using other HTCondor environments. We reached the https://github.com/reanahub/reana-job-controller/tree/master/etc and we found 10_cernsubmit.config and krb5.conf files that look to me that is what we have to modify or extend with our configurations, is that correct? Do we need to install HTCondor packages and daemons to convert our reana on a submit machine?
detailed procedure to change the configuration to configure other Compute Backends using other HTCondor environments
Hello @jordidem. To have your own HTCondor support you would have to subclass the JobManager class, you can find a detailed explanation on how it works in this paper http://cds.cern.ch/record/2696223/files/CERN-IT-2019-004.pdf. Probably there will be many commonalities with the CERN implementation, in that case, we could have a look at it and perhaps create a superclass with the common code?
Do we need to install HTCondor packages and daemons to convert our reana on a submit machine?
Yes, that is the other part that needs extension. As an example, at CERN we build REANA Job Controller with the htcondorcern build alrgument and that includes the necessary packages to submit jobs to CERN HTCondor, in your case you would have to add the instructions to install the packages you need.
resources clause applies to these mounts. (2) If you have /cvmfs/... mounted on your nodes already, then you would basically need to just expose these host mounts to job pods without CSI. This can be done at the cluster creation time, e.g. for development purposes, if I have a node where /cvmfs is mounted, I create cluster as follows: reana-dev cluster-create -m /var/reana:/var/reana -j /cvmfs:/cvmfs. Here, the resources does not have to be used, since any host path could be mounted in a similar way, such as /mybigdata/.... So this second technique isn't CVMFS specific actually. (3) Please tell me which one you would like to try and I can point you to the appropriate code.
@hollowec Sorry for the late reply! Please have a look at the documentation of components.reana_workflow_controller.environment.REANA_JOB_HOSTPATH_MOUNTS in our helm readme and the explanations in our commons code. Basically when you create cluster you should pass something like:
components:
reana_workflow_controller:
environment:
REANA_JOB_HOSTPATH_MOUNTS: '[{"name": "cvmfs", "hostPath": "/cvmfs", "mountPath": "/cvmfs"}]'
REANA_RUNTIME_KUBERNETES_KEEP_ALIVE_JOBS_WITH_STATUSES: failed
image: reanahub/reana-workflow-controllerThis will make /cvmfs path from nodes being available to job pods by host path mounting.
Hello developers! I'm re-posting here a comment by @sinclert on Mattermost. We're really interested on this ongoing bug from NYU
Hi team :wave:
We have been recently trying to run a workflow on your REANA Kubernetes instance, but receiving cryptic logs without any useful debugging information (just node: <...> failed. reason: unknown). This is the reference to the workflow: https://reana.cern.ch/details/d834f61f-3da0-42a6-8606-f5117927539b
Are there any deeper logs you could access as admin, that could help us debug what is going on?
As a hint: the workflow in question was working, on the CERN Kubernetes instance, on the very early versions of REANA 0.7 (0.7.0, 0.7.1...) but eventually it stopped working. Could there be any Kubernetes-level resource policies involved?
Thanks!
I want to set up a minimal reana development environment locally.
According to the docs here: http://docs.reana.io/administration/deployment/deploying-locally/
it recommends to:
Fork and clone all REANA source code repositories:This in turn tries to fork and clone several repositories, including examples
What repositories do I need to clone to minimally setup a local dev env?