These are chat archives for nextflow-io/nextflow

25th
Feb 2019
Rad Suchecki
@rsuchecki
Feb 25 06:33
Someone remind me please how do I execute a jar in a nextflow process script block if the said jar sits under bin/?
Rad Suchecki
@rsuchecki
Feb 25 06:41
ahh ${workflow.projectDir}/bin/foo.jar
Qi ZHAO
@likelet
Feb 25 06:44
@rsuchecki in linux-like system, you can write a wrapper script and put it into the bin folder. Then try call your jar file with that script.
or use java -jar ${baseDir}/bin/foo.jar in your process
Rad Suchecki
@rsuchecki
Feb 25 06:46
:+1:
bahnk
@bahnk
Feb 25 11:37
@pditommaso hi Paolo, I think I solved my problem
it was my mistake actually. I was building my jar file only with the class files, the source code was not included.
i need to do some tests, but it seems to work at the moment
thank you very much
Luca Cozzuto
@lucacozzuto
Feb 25 12:07
Hi all! I'm trying to use nextflow on AWS but I don't understand hot to check the presence of linked file within the workdir structure... :/
sometimes I see that the input is in a tmp folder inside the normale structure
sometimes not
Paolo Di Tommaso
@pditommaso
Feb 25 12:41
there aren't linked files when using batch
Luca Cozzuto
@lucacozzuto
Feb 25 12:43
I know. I see they are in a tmp folder
but sometimes they are not there
but the program see some of them (!)
so I don't know how to debug...
process multiQC {
    publishDir outputMultiQC, mode: 'copy'

    input:
    file ("*") from raw_fastqc_files.mix(logTrimming_for_QC,trimmed_fastqc_files).flatten().collect()
    output:
    file("multiqc_report.html") into multiQC 

    script:
"""
    multiqc .
"""
}
Oren Tzfadia
@ortzf
Feb 25 13:09
@apeltzer after I put them in /bin - how to I evoke them from my nf pipeline?
#!/usr/bin/env Rscript ?
Alexander Peltzer
@apeltzer
Feb 25 13:11
scriptname.R <script-options>
SHould do the trick
The env stuff should be in your R script ideally
Oren Tzfadia
@ortzf
Feb 25 13:12
cool ill try
Jorrit Boekel
@glormph
Feb 25 13:22
Hi all, I have an issue that pops up when analyzing a lot of data (proteomics). I have a pipeline where the input is from a line-based textfile, one input file per line. All is fine until I hit around 1700 input files, where the pipeline starts failing in 10% of cases.
1750-1800, fails on 50%
1900 files - only fails
micans
@micans
Feb 25 13:23
what kind of executor -- is it throttled?
Jorrit Boekel
@glormph
Feb 25 13:23
The error is
N E X T F L O W  ~  version 19.01.0
Launching `quant_proteomics.nf` [nice_bohr] - revision: 788647a300
[warm up] executor > slurm
ERROR ~ Unexpected error [NullPointerException]
Slurm
micans
@micans
Feb 25 13:23
ah ok
Jorrit Boekel
@glormph
Feb 25 13:23
Ah I had not thought of the executor yet.
micans
@micans
Feb 25 13:24
I have no experience with slurm sadly
Jorrit Boekel
@glormph
Feb 25 13:24
the pipeline itself is quite big, ~850 lines without much docs.
I'll try to run local to see if that solves anything.
micans
@micans
Feb 25 13:25
seems unlikely!?
Jorrit Boekel
@glormph
Feb 25 13:25
Indeed, same error
At least if I got a different error I'd get an indication that slurm wasnt cooperating.
The nullpointer points to a line in the script, which is a channel definition.
Paolo Di Tommaso
@pditommaso
Feb 25 13:26
Open an issue on GitHub with the complete log file and a test case
Jorrit Boekel
@glormph
Feb 25 13:27
mzmlfiles
  .buffer(size: amount_mzml.value)
  .map { it.sort( {a, b -> a[1] <=> b[1]}) } // sort on sample for consistent .sh script in -resume
  .map { it -> [it.collect() { it[0] }, it.collect() { it[2] }, it.collect() { it[3] } ] } // lists: [sets], [mzmlfiles], [plates]
  .into { mzmlfiles_all; mzmlfiles_all_count }
Thanks. How would I make the test case? I guess there can be some dummy files since it happens before execution?
Jorrit Boekel
@glormph
Feb 25 13:37
Ah there is a template :)
bahnk
@bahnk
Feb 25 14:38
@pditommaso I did my tests and I still have my problem actually
@pditommaso it's not possible to pass a custom class with output
output:
   set val(my_object), file("*.txt") into new_channel
it gives:
WARN: [test (/path/two)] Unable to resume cached task -- See log file for details
@pditommaso should i open an issue on github?
bahnk
@bahnk
Feb 25 14:47
I mean, maybe it is the expected behaviour. I guess that the Taskprocessor doesn't know my object, so it cannot rebuild it and pass it to a channel
hydriniumh2
@hydriniumh2
Feb 25 15:20
Does anyone know of a way to do unit tests on nextflow processes?
Luca Cozzuto
@lucacozzuto
Feb 25 15:25
Hi @pditommaso. Doing some check reveals that
    file ("*") from raw_fastqc_files.mix(logTrimming_for_QC,trimmed_fastqc_files).flatten().collect()
is not collecting the files in AWS
do you see some errors in this?
Stephen Kelly
@stevekm
Feb 25 18:21
@lebernstein there is a discussion about publishing to a database here: nextflow-io/nextflow#743 ; not sure if you saw this one already, also includes a link to a demo I made on a naive implementation of something like that
also in your workflow.onComplete its also possible to check if the completion was a success or failure, I do that in order to alter the email output here: https://github.com/NYU-Molecular-Pathology/demux-nf/blob/master/main.nf#L372