Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 20 15:38
    dprelipcean closed #201
  • Sep 20 15:38
    dprelipcean commented #201
  • Sep 20 15:21
    tiborsimko commented #201
  • Sep 20 15:06
    dprelipcean opened #201
  • Sep 20 15:03
    dprelipcean closed #195
  • Sep 16 16:24
    tiborsimko closed #200
  • Sep 16 16:24

    tiborsimko on master

    cli: fix auto tagging * All co… release: 0.6.0.dev20190912 (compare)

  • Sep 16 16:11
    tiborsimko closed #254
  • Sep 16 16:11

    tiborsimko on master

    setup: reana-commons version bu… (compare)

  • Sep 16 16:09
    tiborsimko closed #130
  • Sep 16 16:09

    tiborsimko on master

    setup: reana-commons version bu… (compare)

  • Sep 16 15:01
    tiborsimko closed #197
  • Sep 16 15:01

    tiborsimko on master

    htcondor: job monitoring & docs… (compare)

  • Sep 16 14:40
    roksys synchronize #197
  • Sep 16 14:18
    roksys opened #130
  • Sep 16 14:11
    roksys opened #254
  • Sep 16 12:59
    roksys synchronize #197
  • Sep 16 12:46
    tiborsimko closed #129
  • Sep 16 12:46

    tiborsimko on master

    externalbackend: reverts back c… (compare)

  • Sep 16 09:13
    roksys opened #197
Lincoln Bryant
@LincolnBryant
sounds good. then do we start up Reana without the --traefik flag?
it looks like the initialization method is basically just installing the Minikube-oriented helm chart?
BenGalewsky
@BenGalewsky
Yes - there is also some funny business about the persistent volumes if you don’t have CERN ceph available
This is all covered in my post
Lincoln Bryant
@LincolnBryant
got it, thanks! Looks very useful, appreciate it
Rok Roškar
@rokroskar
@diegodelemos @BenGalewsky fwiw tiller will be removed in helm 3.0
BenGalewsky
@BenGalewsky
(although my openstack kubernetes deployment includes NFS storage class) You could use the stable/nfs-server-provisioner helm chart to get read/write many
I have a first draft of a helm chart for reana here:
https://github.com/BenGalewsky/reana-cluster/tree/helm_chart/reana
Haven’t tackled the ingress in that chart yet
Rok Roškar
@rokroskar
awesome!
Diego
@diegodelemos

Error: release reana-traefik failed: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "kube-system"

@LincolnBryant We are deploying Traefik in the kube-system namespace by default. For instance, here at CERN we have already a way of bootstrapping a Kubernetes cluster with Traefik inside so we do not use --traefik for production deployments. However, if you would find it useful to just deploy with reana-cluster init --traefik we can make it configurable and you would use the namespace you use for your REANA instance :)

Lukas
@lukasheinrich
my 2 cents: I think for now helm is closest to what you might call an industry standard
the promise is that existing helm charts will work with helm 3 (tiller-less)
so I think it's a good investment to create a chart
Lincoln Bryant
@LincolnBryant
thanks @diegodelemos . I've got Traefik deployed systemwide now, we'll give it another try. Does Reana use any Traefik-specific features or can another ingress like Nginx be used?
Diego
@diegodelemos

thanks @diegodelemos . I've got Traefik deployed systemwide now, we'll give it another try. Does Reana use any Traefik-specific features or can another ingress like Nginx be used?

In principle you could use any Ingress controller since we just use paths in our Ingress resources

Lukas
@lukasheinrich
dear all -- I'm back at CERN after some useful discussions with the IRIS-HEP folks
if you have time i'd be happy to meet up and discuss a few things
Tibor Šimko
@tiborsimko
@lukasheinrich What about tomorrow morning at some convenient time?
BenGalewsky
@BenGalewsky
@diegodelemos - I think using a different ingress controller would require some annotation changes to the template
BenGalewsky
@BenGalewsky
Made it through ingress customization and now can deploy the helm chart and use my Nginx ingress with a letsencrypt certificate manager. Now trying to start a reana-workflow-engine-serial - the batch-serial pod errors out with
HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /job_cache?job_spec=% …
I’m having trouble puzzling out how the workflow_controller starts up the job and what configuration I’m missing
Diego
@diegodelemos

@diegodelemos - I think using a different ingress controller would require some annotation changes to the template

Yes, you are right. I kind of took this for granted, but this is just to say that the Ingress Controller is implemented by NGINX, on top of that, you might want to add different configurations depending on your needs. At CERN we have, for example, documentation to configure a SSL passthrough with NGINX and looks like this https://clouddocs.web.cern.ch/clouddocs/containers/tutorials/lb.html#ingress-ssl-passthrough (note the kubernetes.io/ingress.class set to nginx and the specific configuration)

Diego
@diegodelemos

Made it through ingress customization and now can deploy the helm chart and use my Nginx ingress with a letsencrypt certificate manager. Now trying to start a reana-workflow-engine-serial - the batch-serial pod errors out with
HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /job_cache?job_spec=% …

@BenGalewsky looks like something went wrong during the initialisation/configuration of REANA Job Controller (RJC).

(1) Are you running latest master? I am guessing you do, since the IP you get for RJC is 0.0.0.0, which would mean you are running RJC as a sidecar of REANA Workflow Engine Serial(R-W-E-Serial). In that case you should first check if the RJC controller container is up and running and there are no errors, something like:

$ kubectl logs batch-serial-aa0ec5ba-781a-43fc-a835-ba3732f5d16f-sdljz job-controller
Serving Flask app "reana_job_controller/app.py"
Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Debug mode: off
2019-07-02 04:13:25,180 | werkzeug | MainThread | INFO |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
2019-07-02 04:13:28,884 | werkzeug | Thread-2 | INFO | 127.0.0.1 - - [02/Jul/2019 04:13:28] "GET /jobs HTTP/1.1" 200 -

(2) If the previous check works, then the problem could be somewhere else. I see the call that times out is /job_cache, so that would be the next place I would look at, this could mean there is a problem with the shared storage since we are checking if the job is cached and that implies reading from shared file system. A quick check would be to go inside the RJC container and check the file system, for example:

$ kubectl exec -ti batch-serial-aa0ec5ba-781a-43fc-a835-ba3732f5d16f-sdljz --container job-controller -- /bin/bash` 
> ls /var/reana
> # test you can read and write
Diego
@diegodelemos

I’m having trouble puzzling out how the workflow_controller starts up the job and what configuration I’m missing

@BenGalewsky regarding the initialisation chain, that goes like follows:

(1) REANA Workflow Controller receives a request to start a workflow.
(2) REANA Workflow Controller creates a Kubernetes job which contains an instance of REANA Workflow Engine _ and REANA Job Controller (here you could have a potential misconfiguration).
(3) REANA Workflow Engine reads the workflow specification and, on each node of the workflow, it submits a job to REANA Job Controller which will be executed by RJC's POSTing to $RJC_ADDRESS/jobs
(4) REANA Job Controller creates a Kubernetes job to execute the workflow step (here you could also have a potential misconfiguration).

Johann Brehmer
@johannbrehmer
Hi all! I would like to cite REANA. I have found the discussion at reanahub/reana#94, but no answer yet. Any ideas? Thanks
Tibor Šimko
@tiborsimko

@johannbrehmer Indeed we don't have a DOI for "REANA-as-the-software" yet. The best would be probably to cite (1) our CHEP2018 paper that describes the REANA platform and the overall architecture in quite some detail:

 \bibitem{ref:chep2018} T.~\v{S}imko,  L.~Heinrich,  H.~Hirvonsalo,  D.~Kousidis,  D.~Rodr\'{\i}guez, ``REANA: A system for reusable research data analyses'',  Computing  in  High  Energy  Physics 2018,  Sofia,  Bulgaria,  9--13 July 2018. \texttt{https://cds.cern.ch/record/2652340} (in press)

(2) or perhaps our Nature Physics opinion piece describing REANA within the fuller analysis preservation and reuse framework:

\bibitem{ref:nature2019} X.~Chen, S.~Dallmeier-Tiessen, R.~Dasler, S.~Feger, P.~Fokianos, J.~.B.~Gonzalez, H.~Hirvonsalo, D.~Kousidis, A.~Lavasa, S.~Mele, D.~Rodr\'{\i}guez, T.~\v{S}imko, T.~Smith, A.~Trisovic, A.~Trzcinska, I.~Tsanaktsidis, M.~Zimmermann, K.~Cranmer, L.~Heinrich, G.~Watts, M.~Hildreth, L.~Lloret~Iglesias, K.~Lassila-Perini, S.~Neubert, ``Open is not enough``, Nature Physics \textbf{15} 113--118 (2019). \texttt{https://www.nature.com/articles/s41567-018-0342-2}
Johann Brehmer
@johannbrehmer
Hi Tibor, thanks a lot!
jordidem
@jordidem

HI @rokroskar ,
As you suggested I'm moving our mail discussion to this channel.

Our main issue is that the first job execution pod for a simple serial workflow example is failing because once the pod is deployed by rancher, the user path which contains the workspace for the workflow execution doesn't contain any file. It looks like the files we uploaded and are visible using reana-client ls, then are not copied/mounted in the pod... In our case this job execution pod is called python2.7

Best,

Rokas Maciulaitis
@roksys
Hi @jordidem, could you paste the output of $ kubectl describe pod <failing pod name>
also logs of job controller, $ kubectl logs <job controller pod name>
Diego
@diegodelemos

Hello @jordidem , something which can be also useful is to check whether all pods share the same storage volume by logging into them.

Before starting with the check, keep in mind that workflows like reana-demo-helloworld run quickly and you might not have time to log into the produced pods, a way of shortcoming this is by increasing the sleeptime when creating the workflow:

$ ~/reana-demo-helloworld/ $ reana-client run -p sleeptime=10000

Once the workflow is submitted, first log into workflow controller:

 $ kubectl exec -ti [workflow-controller-pod-name] bash
> ls -R /var/reana/users/00000000-0000-0000-0000-000000000000
...

In another session, do the same with REANA-Job-Controller:

 $ kubectl exec -ti [job-controller-pod-name] bash
> ls -R /var/reana/users/00000000-0000-0000-0000-000000000000
...

And also check inside the job created by REANA-Job-Controller:

$ # a new job will be spawned
$ kubectl get pods                                                 
NAME                                                      READY   STATUS    RESTARTS   AGE
489637cf-0418-4b0b-9b10-2cd32c245c22-4bxl2                1/1     Running   0          4m2s
batch-serial-1fe96924-2a88-47ac-9613-665c481882f5-gwm9j   2/2     Running   0          4m14s
db-6f8f8579c9-59nv4                                       1/1     Running   0          10m
....
$ kubectl exec -ti 489637cf-0418-4b0b-9b10-2cd32c245c22-4bxl2 bash # replace with your Job UUID
> ls -R /var/reana/users/00000000-0000-0000-0000-000000000000
...

In all cases, you should be able to write a file from one of the pods, and see that the file can be listed/read everywhere else.

jordidem
@jordidem

Hi @roksys ,
kubectl describe pod ce5a1325-2e8c-44c3-a67a-da110eff56bd-hkhdb

Name: ce5a1325-2e8c-44c3-a67a-da110eff56bd-hkhdb
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: reana-server-test/192.168.96.166
Start Time: Mon, 05 Aug 2019 14:30:06 +0200
Labels: controller-uid=c1b241a5-b77c-11e9-bdc3-001a4aab007f
job-name=ce5a1325-2e8c-44c3-a67a-da110eff56bd
Annotations: cni.projectcalico.org/podIP: 10.42.0.250/32
Status: Failed
IP: 10.42.0.250
Controlled By: Job/ce5a1325-2e8c-44c3-a67a-da110eff56bd
Containers:
ce5a1325-2e8c-44c3-a67a-da110eff56bd:
Container ID: docker://4f79a57f85f0395985e10fe639a7bb5132751cf63483df305f7c7ecad896e560
Image: python:2.7
Image ID: docker-pullable://docker.io/python@sha256:d2aaba463a323836d5ad168c1180f22a4351f0f1c732f40d308d8772a39c1be1
Port: <none>
Host Port: <none>
Command:
bash
-c
cd /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/2c7d3d66-42c9-4650-8e84-ec21c4831534 ; python "code/helloworld.py" --inputfile "data/names.txt" --outputfile "results/greetings.txt" --sleeptime 0
State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 05 Aug 2019 14:30:07 +0200
Finished: Mon, 05 Aug 2019 14:30:07 +0200
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/reana/users/00000000-0000-0000-0000-000000000000/workflows/2c7d3d66-42c9-4650-8e84-ec21c4831534 from reana-shared-volume (rw,path="users/00000000-0000-0000-0000-000000000000/workflows/2c7d3d66-42c9-4650-8e84-ec21c4831534")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-hl4zr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
reana-shared-volume:
Type: HostPath (bare host directory volume)
Path: /var/reana
HostPathType:
default-token-hl4zr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-hl4zr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message


Normal Scheduled 9s default-scheduler Successfully assigned default/ce5a1325-2e8c-44c3-a67a-da110eff56bd-hkhdb to reana-server-test
Normal Pulled 8s kubelet, reana-server-test Container image "python:2.7" already present on machine
Normal Created 8s kubelet, reana-server-test Created container
Normal Started 8s kubelet, reana-server-test Started container

kubectl logs job-controller-86687d4d9d-k9wvq

2019-08-05 12:30:06,464 | werkzeug | Thread-542 | INFO | 10.42.0.249 - - [05/Aug/2019 12:30:06] "GET /job_cache?job_spec=%7B%22experiment%22%3A+%22default%22%2C+%22image%22%3A+%22python%3A2.7%22%2C+%22cmd%22%3A+%22+python+%5C%5C%5C%22code%2Fhelloworld.py%5C%5C%5C%22+--inputfile+%5C%5C%5C%22data%2Fnames.txt%5C%5C%5C%22+--outputfile+%5C%5C%5C%22results%2Fgreetings.txt%5C%5C%5C%22+--sleeptime+0+%5C%22%22%2C+%22prettified_cmd%22%3A+%22python+%5C%22code%2Fhelloworld.py%5C%22+--inputfile+%5C%22data%2Fnames.txt%5C%22+--outputfile+%5C%22results%2Fgreetings.txt%5C%22+--sleeptime+0%22%2C+%22workflow_workspace%22%3A+%22%2Fvar%2Freana%2Fusers%2F00000000-0000-0000-0000-000000000000%2Fworkflows%2F2c7d3d66-42c9-4650-8e84-ec21c4831534%22%2C+%22job_name%22%3A+%22python+%5C%22code%2Fhelloworld.py%5C%22+--inputfile+%5C%22data%2Fnames.txt%5C%22+--outputfile+%5C%22results%2Fgreetings.txt%5C%22+--sleeptime+0%22%2C+%22cvmfs_mounts%22%3A+%22false%22%2C+%22workflow_uuid%22%3A+%222c7d3d66-42c9-4650-8e84-ec21c4831534%22%7D&workflow_json=%7B%22environment%22%3A+%22python%3A2.7%22

Rokas Maciulaitis
@roksys
@jordidem, hm I don't see anything strange, everything looks fine to me with the mounts
could you try to run hello world with increased sleep time (as Diego suggested above) and connect to the job pod?
jordidem
@jordidem
@roksys We changed the sleeptime to 1000 and the problem is the same. The job can't see /var/reana. Since we are using reana-cluster 0.5, with a LOCAL deployment and not using Ceph at all, we downgraded kubernetes client to version 9, and accordingly now we've downgraded the rancher kubernetes version to 13.7 using the following compatibility table (https://github.com/kubernetes-client/python#compatibility) Any idea?
Diego
@diegodelemos

@roksys We changed the sleeptime to 1000 and the problem is the same. The job can't see /var/reana. Since we are using reana-cluster 0.5, with a LOCAL deployment and not using Ceph at all, we downgraded kubernetes client to version 9, and accordingly now we've downgraded the rancher kubernetes version to 13.7 using the following compatibility table (https://github.com/kubernetes-client/python#compatibility) Any idea?

I think at this point what makes more sense is to check the Rancher deployment itself, what's the architecture behind it. Which documentation did you follow to deploy Rancher?

Note that the LOCAL deployment relies on the fact that all containers run on the same host. Therefore, if there is a volume mount from container 1 to host and from container 2 to host being the host path /var/reana, then the following should be true: container 1 writes echo 'hello' > /var/reana/test and container 2 can access the file and content cat /var/reana/test.

For example, if you have followed https://rancher.com/blog/2018/2018-05-18-how-to-run-rancher-2-0-on-your-desktop/, seems relevant to see whether Docker-for-desktop can work with hostPath's which, for instance, needs some configuration in the case of Mac, see https://forums.docker.com/t/how-to-mount-hostpath-using-docker-for-mac-kubernetes/44083

jordidem
@jordidem
Hi @Diego, thanks for the quick answer. Our architecture behind is a virtual-server with CentOS 7. We have installed rancher using the rancher/rancher:latest from the docker registry. On the other hand, in the same server we have installed the virtual environment for the reana-cluster. Nevertheless, if we clone the failed job and modify the mount_path by hand, the job runs and ends correctly.
Diego
@diegodelemos
@jordidem could you describe a bit more what you had to do? For example, what was the previous value of the mount_path and what you had to use instead? Sounds like is something that could help other people who want to run REANA on Rancher
jordidem
@jordidem

Sorry for the long message. We tried with two different tests. One running whoami and sleep commands, and the other one based on the helloworld.py example.

In both cases, after job-serial starts, mount volume are failing with:

Events:
  Type     Reason  Age                    From                        Message
  ----     ------  ----                   ----                        -------
  Normal   Pulled  8m15s (x281 over 68m)  kubelet, reana-server-test  Container image "python:2.7-slim" already present on machine
  Warning  Failed  3m13s (x304 over 68m)  kubelet, reana-server-test  Error: stat /var/reana: no such file or directory

1) For the whoami example:
If we clone this job to a POD and manually modify de "subPath" mount point, the job ends correctly.

Events:
  Type    Reason     Age                 From                        Message
  ----    ------     ----                ----                        -------
  Normal  Scheduled  32m                 default-scheduler           Successfully assigned default/test-lzz2n to reana-server-test
  Normal  Pulled     118s (x3 over 32m)  kubelet, reana-server-test  Container image "python:2.7-slim" already present on machine
  Normal  Created    117s (x3 over 32m)  kubelet, reana-server-test  Created container test
  Normal  Started    117s (x3 over 32m)  kubelet, reana-server-test  Started container test

Volume Name: reana-shared-volume
mount_path: /var/reana
mount_point: /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861
subPath in volume: users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861

Modification:

Volume Name: reana-shared-volume
mount_path: /var/reana
mount_point: /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861
subPath in volume: ""

Inside the "cloned" pod:

root@test-6ssgs: mount | grep reana
/dev/md2 on /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861 type ext4 (rw,relatime,data=ordered)

root@test-6ssgs:/var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861# ls
db  users

2) For the helloworld.py example:
The job fails because is trying to access to code files on a bad mount_point. So, the entrypoint command try this:

Entrypoint

bash,-c,cd /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861 ; python "code/helloworld.py" --inputfile "data/names.txt" --
outputfile "results/greetings.txt" --sleeptime 60

But as we see in the POD, the path looks like this :

/var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861/users/00000000-0000-0000-0000-000000000000/workflows/
ce416b12-1feb-445b-85b4-2bcf7b28b861

If we modify the "cd" commnad with absolute path it works, but this is not a real solution:

- command:
  - bash
  - -c
  - 'cd /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861/users/00000000-0000-0000-0000-000000000000/workflows/
  ce416b12-1feb-445b-85b4-2bcf7b28b861
    ; python "code/helloworld.py" --inputfile "data/names.txt" --outputfile "results/greetings.txt"
    --sleeptime 90

$ reana-client ls
NAME                    SIZE   LAST-MODIFIED      
data/names.txt          20     2019-08-07T10:42:05
results/greetings.txt   1      2019-08-07T10:49:06
code/helloworld.py      2899   2019-08-07T10:42:04

Instead of modifying the entrypoint, we modify the mount_point to /var/reana and subPath to "", it works withou modifying the entrypoint. This is because all workflows are accessible since the whole users path is mounted in the POD, but againg this is so .
Volume Name: reana-shared-volume
mount_path: /var/reana
mount_point: /var/reana
subPath in volume: ""

Rokas Maciulaitis
@roksys
root@test-6ssgs:/var/reana/users/00000000-0000-0000-0000-000000000000/workflows/ce416b12-1feb-445b-85b4-2bcf7b28b861# ls db users this part is really odd
Diego
@diegodelemos

@jordidem Looks like subPath is not working as it should be... This is a Kubernetes feature we are using in REANA so we can use always the same root path and then select, using subPath, which directory a specific workflow should see (only its own workspace).

TL;DR not using subPath causes a security vulnerability which allows workflows to access files and directories outside their workspace, which in other words, would mean user X can delete all user Y data by running a simple rm. If you are interested, we provide hereafter a pure Kubernetes example of how this happens.

  1. Create a test.yaml file with the following content:
apiVersion: v1
kind: Pod
metadata:
  name: database
spec:
    containers:
    - name: database
      image: busybox
      command:
      - /bin/sh
      - -c
      - "sleep 1000"
      volumeMounts:
      - mountPath: /var/db
        name: database-dir
        subPath: database
    volumes:
    - name: database-dir
      hostPath:
        path: /tmp
  1. Then we create the database directory inside /tmp in the machine where the pods run. In our case that's minikube:
$ minikube ssh
> mkdir /tmp/database
> echo 'database stuff' > /tmp/database/data.txt
> # and we create something else inside /tmp which
> # the pod shouldn't have access to
> mkdir /tmp/somethingelse
> echo 'some,other,stuff' > /tmp/somethingelse/data.csv
  1. Once this is done, you can create the pod:

    $ kubectl create -f test.yaml
  2. Log into the pod, and actually verify that you can only access "your files":

$ kubectl exec -ti database sh
/ # cat /var/db/data.txt 
database stuff
/ # cat /var/../somethingelse/data.csv
cat: can't open '/var/../somethingelse/data.csv': No such file or directory
  1. Now, trying without subPath, we copy the file:
$ cp test.yaml test-vulnerable.yaml
  1. We remove the subPath and rename the pod:
apiVersion: v1
kind: Pod
metadata:
-  name: database
+  name: database-vulnerable
spec:
    containers:
-    - name: database
+    - name: database-vulnerable
      image: busybox
      command:
      - /bin/sh
      - -c
      - "sleep 1000"
      volumeMounts:
      - mountPath: /var/db/
        name: database-dir
-        subPath: database
+        subPath: ""
    volumes:
    - name: database-dir
      hostPath:
        path: /tmp
  1. And we can see now, that different pods (jobs in the REANA vocabulary) would see outside their working space.
$ kubectl exec -ti database-vulnerable sh
/ # cat /var/db/database/data.txt
database stuff
/ # cat /var/db/somethingelse/data.csv                         
some,other,stuff
/ # echo 1 > /var/db/somethingelse/data.csv 
/ # cat /var/db/somethingelse/data.csv 
1
Rokas Maciulaitis
@roksys
Rokas Maciulaitis
@roksys
It is seems to be a rancher issue
would it be possible for you to use another deployment type?
jordidem
@jordidem

Dear all, finally we managed to solve the issue using Rancher with the following workaround extracted from coment of gitlawr on rancher/rancher#14836 . Literally we did the following steps

1) Edit cluster, Edit as YAML
2) Add the following flags for kubelet:

services:
   kubelet:
        extra_args:
            containerized: "true"
        extra_binds: 
             - "/:/rootfs:rshared"

3) Click save and wait till the cluster is updated.

Notes:
The community is planning to deprecate the "--containerized" for kubelet(kubernetes/kubernetes#74148).
But the flag is essential for the capability as there is no alternative at the moment.

Now our REANA deployment on Rancher works! Yihaaa!
Thanks for the help and pointing the link!

Diego
@diegodelemos
@jordidem nice finding! would you like to share the solution on the reanahub/reana-cluster#117? I think it would be really useful for people who might be in the same situation as you were :)
Oriol
@zakeruga
Hi Reana team, I’m a co-worker of @jordidem . I have a question, ¿can I have another extra volume like /var/reana but visible for all the job pods?
The reason why I need this is because I have a lot of big files (GB) as input and I don’t want to replicate this files for each user’s workflow.
Rokas Maciulaitis
@roksys
Hi @zakeruga, unfortunately there is not that kind of feature. Job-Controller mounts only workspace (/var/reana/users/<user_id>/workflows/<workflow_id>) for the job pods.
Rokas Maciulaitis
@roksys
We have this feature in Master, but it is not released - reanahub/reana-job-controller@ed125c0