These are chat archives for nextflow-io/nextflow

24th
Sep 2018
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:15
No, nextflow is not cwl
misssoft
@misssoft
Sep 24 2018 08:47
@pditommaso still struggle a bit with k8s, having the exact problem like issue #833 (nextflow-io/nextflow#833), run the command:
docker@ubuntu-docker:~$ nextflow kuberun https://github.com/oxfordmmm/rnaseq-nf -v "azurefile5t:/mnt/nextflow" -profile azurekube
Launcher pod spec file: .nextflow.pod.yaml
Pod started: deadly-woese
container_linux.go:247: starting container process caused "chdir to cwd (\"/mnt/nextflow/docker\") set in config.json failed: no such file or directory"
The pod is created but in a state of ContainerCannotRun
Here is the pod yaml, anything suspicious?
docker@ubuntu-docker:~$ more .nextflow.pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: deadly-woese
namespace: default
labels: {app: nextflow, runName: deadly-woese}
spec:
restartPolicy: Never
containers:
  • name: deadly-woese
    image: nextflow/nextflow:0.31.1
    command: [/bin/bash, -c, 'source /etc/nextflow/init.sh; nextflow run https://github.com/oxfordmmm/rnaseq-nf
    -name deadly-woese']
    workingDir: /mnt/nextflow/docker
    env:
    • {name: NXF_WORK, value: /mnt/nextflow/docker/work}
    • {name: NXF_ASSETS, value: /mnt/nextflow/projects}
    • {name: NXF_EXECUTOR, value: k8s}
      volumeMounts:
    • {name: vol-1, mountPath: /mnt/nextflow}
    • {name: vol-2, mountPath: /etc/nextflow}
      volumes:
  • name: vol-1
    persistentVolumeClaim: {claimName: azurefile5t}
  • name: vol-2
    configMap: {name: nf-config-dc811795}
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:51
please, format code using triple `
misssoft
@misssoft
Sep 24 2018 08:52
yes, sorry.
docker@ubuntu-docker:~$ more .nextflow.pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: deadly-woese
  namespace: default
  labels: {app: nextflow, runName: deadly-woese}
spec:
  restartPolicy: Never
  containers:
  - name: deadly-woese
    image: nextflow/nextflow:0.31.1
    command: [/bin/bash, -c, 'source /etc/nextflow/init.sh; nextflow run https://github.com/oxfordmmm/rnaseq-nf
        -name deadly-woese']
    workingDir: /mnt/nextflow/docker
    env:
    - {name: NXF_WORK, value: /mnt/nextflow/docker/work}
    - {name: NXF_ASSETS, value: /mnt/nextflow/projects}
    - {name: NXF_EXECUTOR, value: k8s}
    volumeMounts:
    - {name: vol-1, mountPath: /mnt/nextflow}
    - {name: vol-2, mountPath: /etc/nextflow}
  volumes:
  - name: vol-1
    persistentVolumeClaim: {claimName: azurefile5t}
  - name: vol-2
    configMap: {name: nf-config-dc811795}
docker@ubuntu-docker:~$ nextflow kuberun https://github.com/oxfordmmm/rnaseq-nf -v "azurefile5t:/mnt/nextflow" -profile azurekube
Launcher pod spec file: .nextflow.pod.yaml
Pod started: deadly-woese
container_linux.go:247: starting container process caused "chdir to cwd (\"/mnt/nextflow/docker\") set in config.json failed: no such file or directory"
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:54
apparently can't create or can't write in the mount path
misssoft
@misssoft
Sep 24 2018 08:55
yes, looks like permission, but how to solve it?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:57
likely creating that path in the target volume
misssoft
@misssoft
Sep 24 2018 08:57
I have manually created the path, using the root in all 3 nodes of the cluster
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:01
it may depend on specific security context or namespace setting for your deployment environment
you will need to investigate about that
misssoft
@misssoft
Sep 24 2018 09:05
Thank you, @pditommaso , I can manually created a pod using the pvc, I will look into that yaml to see any differences with the yaml created by NF...
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:06
any feedback to improve it is welcome
misssoft
@misssoft
Sep 24 2018 09:06
Will do!
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:06
:ok_hand:
Anthony Underwood
@aunderwo
Sep 24 2018 11:10
image.png
Making some progress with AWS batch but when I run nextflow it hangs after the 1st few submissions
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:10
very good!
Anthony Underwood
@aunderwo
Sep 24 2018 11:11
in AWS the job has the mysterious :
image.png
But nextflow is still running waiting for those jobs to complete
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:12
all of them ?
Anthony Underwood
@aunderwo
Sep 24 2018 11:13
yes it's submitted the 1st 4 that it can within the process DAG and these all FAIL as reported by AWS but the nextflow master process is still alive waiting for them to complete
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:14
check the job log in aws dashboard
Anthony Underwood
@aunderwo
Sep 24 2018 11:15
image.png
Flowcraft still thinks they are running
Ah ha I have 4 that are runnable. What might these be waiting for?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:18
likely .. :smile:
Anthony Underwood
@aunderwo
Sep 24 2018 11:20
Waiting for spot instance???? I have no instances currently showing up as running in the EC2 dashboard
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:21
check spot instances requests
Anthony Underwood
@aunderwo
Sep 24 2018 11:22
image.png
Need some other role
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:23
still to work on that Batch config :D
Anthony Underwood
@aunderwo
Sep 24 2018 11:23
yeah :)
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:24
follow the aws docs for a simple batch job without NF
when you are use it works, move to NF
Anthony Underwood
@aunderwo
Sep 24 2018 11:25
:+1:
Anthony Underwood
@aunderwo
Sep 24 2018 11:57

Making progress and jobs are running but but am now seeing

download failed: s3://xxxxxxxxx/workdir/17/c3fa2d15068bfdb71c18625e9f549f/.command.run to - An error occurred (403) when calling the HeadObject operation: Forbidden

Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:59
s3 permissions ?
Alexander Peltzer
@apeltzer
Sep 24 2018 11:59
Looks like it yes
and the other problem probably th emissing permissions for requesting SpotPrice Instances
Anthony Underwood
@aunderwo
Sep 24 2018 12:00
image.png
My IAM role has full S3 permissions
I needed 3 policies in the AmazonEC2SpotFleetRole - the one that was missing was AmazonEC2SpotFleetAutoscaleRole
Paolo Di Tommaso
@pditommaso
Sep 24 2018 12:04
what about the Instance role?
it must have S3 permissions
Anthony Underwood
@aunderwo
Sep 24 2018 12:06
ah ha no it only had AmazonEC2ContainerServiceforEC2Role
I'll try that
Alexander Peltzer
@apeltzer
Sep 24 2018 12:10
@aunderwo if there is stuff missing in my blogpost let me know please
Trying to make it as complete as possible
Anthony Underwood
@aunderwo
Sep 24 2018 12:11
@apeltzer Sure - I'll try and capture what worked for me. Without the blog though I would bot have known where to start :) So BIG thanks
Nextflow batch docs also v helpful. Thanks @pditommaso
Alexander Peltzer
@apeltzer
Sep 24 2018 12:11
Glad that helps :-)
Maxime Garcia
@MaxUlysse
Sep 24 2018 12:18
@aunderwo Make your own blog post too ;-)
Paolo Di Tommaso
@pditommaso
Sep 24 2018 12:18
good idea ! :smile:
Alexander Peltzer
@apeltzer
Sep 24 2018 12:20
Yup - or that :-)
Anthony Underwood
@aunderwo
Sep 24 2018 13:29
image.png
My nextflow processes are now completing successfully BUT the batch jobs are reporting as failing
full S3 permissions have been given
Anthony Underwood
@aunderwo
Sep 24 2018 13:39
When an AWS instance is created using an AMI image, I presume the instance will not have the instance type of the instance from which the image was made with but will have RAM, CPUs of the instance type selected??
Bo
@wenbostar
Sep 24 2018 14:23
@pditommaso , thank you for your answer. Is there any tool to convert nextflow workflow to cwl workflow?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 14:28
nope
Bo
@wenbostar
Sep 24 2018 14:30
Got it. Thanks.
Alexander Peltzer
@apeltzer
Sep 24 2018 14:56
@anthony yes, you can create an AMI with a small t2.micro for example and later deploy a large one with it - doesn’t matter
Vladimir Kiselev
@wikiselev
Sep 24 2018 16:07

maybe we can get help here quickly: today we tried to start 2 new pipelines and received the same errors for both of them:

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

is it familiar to people? Also no .exitcode is produced. I feel it must be something simple...

Evan Floden
@evanfloden
Sep 24 2018 16:09
Is this issue related?
micans
@micans
Sep 24 2018 17:07
Thanks @evanfloden , it turned out that we missed k8s+NF-specific config settings. The error is slightly surprising, but no matter.