These are chat archives for nextflow-io/nextflow

24th
Sep 2018
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:15 UTC
No, nextflow is not cwl
misssoft
@misssoft
Sep 24 2018 08:47 UTC
@pditommaso still struggle a bit with k8s, having the exact problem like issue #833 (nextflow-io/nextflow#833), run the command:
docker@ubuntu-docker:~$ nextflow kuberun https://github.com/oxfordmmm/rnaseq-nf -v "azurefile5t:/mnt/nextflow" -profile azurekube
Launcher pod spec file: .nextflow.pod.yaml
Pod started: deadly-woese
container_linux.go:247: starting container process caused "chdir to cwd (\"/mnt/nextflow/docker\") set in config.json failed: no such file or directory"
The pod is created but in a state of ContainerCannotRun
Here is the pod yaml, anything suspicious?
docker@ubuntu-docker:~$ more .nextflow.pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: deadly-woese
namespace: default
labels: {app: nextflow, runName: deadly-woese}
spec:
restartPolicy: Never
containers:
  • name: deadly-woese
    image: nextflow/nextflow:0.31.1
    command: [/bin/bash, -c, 'source /etc/nextflow/init.sh; nextflow run https://github.com/oxfordmmm/rnaseq-nf
    -name deadly-woese']
    workingDir: /mnt/nextflow/docker
    env:
    • {name: NXF_WORK, value: /mnt/nextflow/docker/work}
    • {name: NXF_ASSETS, value: /mnt/nextflow/projects}
    • {name: NXF_EXECUTOR, value: k8s}
      volumeMounts:
    • {name: vol-1, mountPath: /mnt/nextflow}
    • {name: vol-2, mountPath: /etc/nextflow}
      volumes:
  • name: vol-1
    persistentVolumeClaim: {claimName: azurefile5t}
  • name: vol-2
    configMap: {name: nf-config-dc811795}
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:51 UTC
please, format code using triple `
misssoft
@misssoft
Sep 24 2018 08:52 UTC
yes, sorry.
docker@ubuntu-docker:~$ more .nextflow.pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: deadly-woese
  namespace: default
  labels: {app: nextflow, runName: deadly-woese}
spec:
  restartPolicy: Never
  containers:
  - name: deadly-woese
    image: nextflow/nextflow:0.31.1
    command: [/bin/bash, -c, 'source /etc/nextflow/init.sh; nextflow run https://github.com/oxfordmmm/rnaseq-nf
        -name deadly-woese']
    workingDir: /mnt/nextflow/docker
    env:
    - {name: NXF_WORK, value: /mnt/nextflow/docker/work}
    - {name: NXF_ASSETS, value: /mnt/nextflow/projects}
    - {name: NXF_EXECUTOR, value: k8s}
    volumeMounts:
    - {name: vol-1, mountPath: /mnt/nextflow}
    - {name: vol-2, mountPath: /etc/nextflow}
  volumes:
  - name: vol-1
    persistentVolumeClaim: {claimName: azurefile5t}
  - name: vol-2
    configMap: {name: nf-config-dc811795}
docker@ubuntu-docker:~$ nextflow kuberun https://github.com/oxfordmmm/rnaseq-nf -v "azurefile5t:/mnt/nextflow" -profile azurekube
Launcher pod spec file: .nextflow.pod.yaml
Pod started: deadly-woese
container_linux.go:247: starting container process caused "chdir to cwd (\"/mnt/nextflow/docker\") set in config.json failed: no such file or directory"
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:54 UTC
apparently can't create or can't write in the mount path
misssoft
@misssoft
Sep 24 2018 08:55 UTC
yes, looks like permission, but how to solve it?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 08:57 UTC
likely creating that path in the target volume
misssoft
@misssoft
Sep 24 2018 08:57 UTC
I have manually created the path, using the root in all 3 nodes of the cluster
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:01 UTC
it may depend on specific security context or namespace setting for your deployment environment
you will need to investigate about that
misssoft
@misssoft
Sep 24 2018 09:05 UTC
Thank you, @pditommaso , I can manually created a pod using the pvc, I will look into that yaml to see any differences with the yaml created by NF...
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:06 UTC
any feedback to improve it is welcome
misssoft
@misssoft
Sep 24 2018 09:06 UTC
Will do!
Paolo Di Tommaso
@pditommaso
Sep 24 2018 09:06 UTC
:ok_hand:
Anthony Underwood
@aunderwo
Sep 24 2018 11:10 UTC
image.png
Making some progress with AWS batch but when I run nextflow it hangs after the 1st few submissions
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:10 UTC
very good!
Anthony Underwood
@aunderwo
Sep 24 2018 11:11 UTC
in AWS the job has the mysterious :
image.png
But nextflow is still running waiting for those jobs to complete
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:12 UTC
all of them ?
Anthony Underwood
@aunderwo
Sep 24 2018 11:13 UTC
yes it's submitted the 1st 4 that it can within the process DAG and these all FAIL as reported by AWS but the nextflow master process is still alive waiting for them to complete
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:14 UTC
check the job log in aws dashboard
Anthony Underwood
@aunderwo
Sep 24 2018 11:15 UTC
image.png
Flowcraft still thinks they are running
Ah ha I have 4 that are runnable. What might these be waiting for?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:18 UTC
likely .. :smile:
Anthony Underwood
@aunderwo
Sep 24 2018 11:20 UTC
Waiting for spot instance???? I have no instances currently showing up as running in the EC2 dashboard
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:21 UTC
check spot instances requests
Anthony Underwood
@aunderwo
Sep 24 2018 11:22 UTC
image.png
Need some other role
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:23 UTC
still to work on that Batch config :D
Anthony Underwood
@aunderwo
Sep 24 2018 11:23 UTC
yeah :)
Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:24 UTC
follow the aws docs for a simple batch job without NF
when you are use it works, move to NF
Anthony Underwood
@aunderwo
Sep 24 2018 11:25 UTC
:+1:
Anthony Underwood
@aunderwo
Sep 24 2018 11:57 UTC

Making progress and jobs are running but but am now seeing

download failed: s3://xxxxxxxxx/workdir/17/c3fa2d15068bfdb71c18625e9f549f/.command.run to - An error occurred (403) when calling the HeadObject operation: Forbidden

Paolo Di Tommaso
@pditommaso
Sep 24 2018 11:59 UTC
s3 permissions ?
Alexander Peltzer
@apeltzer
Sep 24 2018 11:59 UTC
Looks like it yes
and the other problem probably th emissing permissions for requesting SpotPrice Instances
Anthony Underwood
@aunderwo
Sep 24 2018 12:00 UTC
image.png
My IAM role has full S3 permissions
I needed 3 policies in the AmazonEC2SpotFleetRole - the one that was missing was AmazonEC2SpotFleetAutoscaleRole
Paolo Di Tommaso
@pditommaso
Sep 24 2018 12:04 UTC
what about the Instance role?
it must have S3 permissions
Anthony Underwood
@aunderwo
Sep 24 2018 12:06 UTC
ah ha no it only had AmazonEC2ContainerServiceforEC2Role
I'll try that
Alexander Peltzer
@apeltzer
Sep 24 2018 12:10 UTC
@aunderwo if there is stuff missing in my blogpost let me know please
Trying to make it as complete as possible
Anthony Underwood
@aunderwo
Sep 24 2018 12:11 UTC
@apeltzer Sure - I'll try and capture what worked for me. Without the blog though I would bot have known where to start :) So BIG thanks
Nextflow batch docs also v helpful. Thanks @pditommaso
Alexander Peltzer
@apeltzer
Sep 24 2018 12:11 UTC
Glad that helps :-)
Maxime Garcia
@MaxUlysse
Sep 24 2018 12:18 UTC
@aunderwo Make your own blog post too ;-)
Paolo Di Tommaso
@pditommaso
Sep 24 2018 12:18 UTC
good idea ! :smile:
Alexander Peltzer
@apeltzer
Sep 24 2018 12:20 UTC
Yup - or that :-)
Anthony Underwood
@aunderwo
Sep 24 2018 13:29 UTC
image.png
My nextflow processes are now completing successfully BUT the batch jobs are reporting as failing
full S3 permissions have been given
Anthony Underwood
@aunderwo
Sep 24 2018 13:39 UTC
When an AWS instance is created using an AMI image, I presume the instance will not have the instance type of the instance from which the image was made with but will have RAM, CPUs of the instance type selected??
Bo
@wenbostar
Sep 24 2018 14:23 UTC
@pditommaso , thank you for your answer. Is there any tool to convert nextflow workflow to cwl workflow?
Paolo Di Tommaso
@pditommaso
Sep 24 2018 14:28 UTC
nope
Bo
@wenbostar
Sep 24 2018 14:30 UTC
Got it. Thanks.
Alexander Peltzer
@apeltzer
Sep 24 2018 14:56 UTC
@anthony yes, you can create an AMI with a small t2.micro for example and later deploy a large one with it - doesn’t matter
Vladimir Kiselev
@wikiselev
Sep 24 2018 16:07 UTC

maybe we can get help here quickly: today we tried to start 2 new pipelines and received the same errors for both of them:

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

is it familiar to people? Also no .exitcode is produced. I feel it must be something simple...

Evan Floden
@evanfloden
Sep 24 2018 16:09 UTC
Is this issue related?
Stijn van Dongen
@micans
Sep 24 2018 17:07 UTC
Thanks @evanfloden , it turned out that we missed k8s+NF-specific config settings. The error is slightly surprising, but no matter.