These are chat archives for nextflow-io/nextflow

23rd
Aug 2018
Sven F.
@sven1103
Aug 23 2018 06:55
@pditommaso I am observing test failures for https://github.com/nextflow-io/nextflow/blob/cd8dfd6bd1cf95be3a3305daa4a4e7bf87978616/src/test/groovy/nextflow/script/WorkflowMetadataTest.groovy#L91, which raises an Assertion Error, because the mocked ScriptRunner is null. I thought this was due to my implementation #841 , so I reverted and tested again, Assertion Error remains. Any ideas why the Mock is null, although it was set up previously in the given: section?
welcome back, btw!
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:38
@apeltzer very nice blog post Alex!
@sven1103 is failing locally or in a travis build ?
welcome back, btw!
Sven F.
@sven1103
Aug 23 2018 08:39
@pditommaso both
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:40
thanks, tho I;m not yet fully recovered .. :satisfied:
Sven F.
@sven1103
Aug 23 2018 08:40
:joy:
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:40
umm, looks weird it's fine on my side, so why it's failing if you rollback the previous implementation ?
Sven F.
@sven1103
Aug 23 2018 08:41
hmmm, strange
i will pull again from master
and check
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:41
let me check the travis build
Sven F.
@sven1103
Aug 23 2018 08:42
kk
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:43
not very useful, do you have the error report when testing locally ?
Sven F.
@sven1103
Aug 23 2018 08:44
sure, one sec
Assertion failed: 

assert object != null
       |
       null

    at nextflow.config.Manifest.<init>(Manifest.groovy:41)
    at nextflow.script.WorkflowMetadata.<init>(WorkflowMetadata.groovy:233)
    at nextflow.script.WorkflowMetadataTest.should populate workflow object(WorkflowMetadataTest.groovy:91)
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:45
ah, make sense
Sven F.
@sven1103
Aug 23 2018 08:45
ahhh
i see it now too
damn it
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:45
this.manifest = new Manifest(owner.session.config.manifest as Map)
Sven F.
@sven1103
Aug 23 2018 08:45
sorry for bothering you -.-
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:45
no problem
Sven F.
@sven1103
Aug 23 2018 08:45
i ll fix it
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:46
I would add a ?: Collections.emptyMap()
Sven F.
@sven1103
Aug 23 2018 08:46
jupp
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:47
also add a version attribute in the manifest (instead of pipelineVersion)
Sven F.
@sven1103
Aug 23 2018 08:47
k

I would add a ?: Collections.emptyMap()

just to make sure: instead of the assertion?

in the Manifest class?
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:49
mmm
I would say in the invocation
Sven F.
@sven1103
Aug 23 2018 08:51
this.manifest = new Manifest((owner.session.config.manifest as Map) ?: Collections.emptyMap()) ?
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:52
yep
Sven F.
@sven1103
Aug 23 2018 08:52
cool cool
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:52
note that Collections.emptyMap is a method
Sven F.
@sven1103
Aug 23 2018 08:52
will fix it right away
hupy
Paolo Di Tommaso
@pditommaso
Aug 23 2018 08:53
I think it's Collections.emptyMap()
Sven F.
@sven1103
Aug 23 2018 08:53
quick typo
i was just sloppy
Paolo Di Tommaso
@pditommaso
Aug 23 2018 09:25
@mes5k I've read only now about your question regarding K8s, that directory is supposed to be created by NF, if it does not it could be a K8s/storage permission problem. However that paths can be defined in the config file. See userDir, projectDir, workDir in the docs.
@mes5k if you want to share more details about your setup I would be happy to include in the doc or share in a post
Karin Lagesen
@karinlag
Aug 23 2018 11:09
I found the following in the config file of a nf pipeline
cpus = { check_max( 2, 'cpus' ) }
how does that one work?
Alexander Peltzer
@apeltzer
Aug 23 2018 11:19
There is a script in the nextflow.config that checks how much CPUs a single node has (written in the variabel max_cpus)
(for example)
This function ensures that you don't take more than available CPUs on the system, when a process is re-scheduled, e.g. because it used the entire defined wall time on a cluster
Karin Lagesen
@karinlag
Aug 23 2018 11:48
ah!
I thought it was a nextflow native thingy ")
also, had no idea you could include more than one config files like that, that solves another problem I have
reading other people's code is quite useful
Alexander Peltzer
@apeltzer
Aug 23 2018 11:59
It is - still learning a lot and finding new ways to do things :-)
Ghost
@ghost~598345d2d73408ce4f6ff925
Aug 23 2018 14:51

I am trying to run nextflow with a docker image, however the docker image contains a conda environment that needs to be activated before running Nextflow. I'm not sure if this specific situation is the cause of an error I can't resolve:

The error:
python: can't open file 'path/to/script.py': [Errno 2] No such file or directory

This normally would be a simple error to fix but I think it's something with the way I've set up the configuration file.

process.container = 'pipeliner/images:latest'
docker {
    sudo=true
    enabled=true
    legacy='source activate pipeliner'
}

Also I can confirm that running without docker and using the conda environment works completely.

Does anyone have suggestions that might point me in the right direction?

Paolo Di Tommaso
@pditommaso
Aug 23 2018 14:53
legacy='source activate pipeliner' makes no sense, it's a flag true|false
it's a custom docker image ie. you can change it ?
Ghost
@ghost~598345d2d73408ce4f6ff925
Aug 23 2018 14:55
Ah, mixed up the variables. Yes it's a custom image. Is there a way to ensure the conda environment is activated?
Paolo Di Tommaso
@pditommaso
Aug 23 2018 14:56
yes, install your stuff in the root env
Ghost
@ghost~598345d2d73408ce4f6ff925
Aug 23 2018 15:29

The root directory for my miniconda install is called "base" but either way, I installed the packages there. This configuration is activating the environment now, however the problem still persists.

process.container = 'pipeliner/images:latest'
docker {
    enabled = true
    runOptions = 'pipeliner/images:latest source activate base'
}

Maybe it's not a conda environment issue and more of a file sharing issue? Here is the full error:

ERROR ~ Error executing process > 'pre_fastqc (ggal_gamma)'

Caused by:
  Process `pre_fastqc (ggal_gamma)` terminated with an error exit status (127)

Command executed [/Users/anthonyfederico/Work/pipeliner/pipelines/templates/fastqc/paired.sh]:

  fastqc --quiet --outdir . /Users/anthonyfederico/Work/pipeliner/pipelines/toy_data/rnaseq/reads/ggal_gamma_1.fq.gz /Users/anthonyfederico/Work/pipeliner/pipelines/toy_data/rnaseq/reads/ggal_gamma_2.fq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: source: No such file or directory

Work dir:
  /Users/anthonyfederico/Work/pipeliner/pipelines/work/8c/e27da49188d268273df343c70bc789
Paolo Di Tommaso
@pditommaso
Aug 23 2018 15:30
this makes no sense runOptions = 'pipeliner/images:latest source activate base'
another option is that you add the env path to the container PATH
have a look here
Ghost
@ghost~598345d2d73408ce4f6ff925
Aug 23 2018 18:05
Still the same problem, the docker image can't find files on my local machine. Possibly due to the way I setup Docker? I'll do some more investigative work on that end.