These are chat archives for nextflow-io/nextflow

11th
Sep 2018
Crabime
@Crabime
Sep 11 2018 02:24
recently i have been testing NF memory usage. Here is my NF scripts snippetprocess sra2fq { tag "$sample.baseName" executor = 'sge' penv 'smp' cpus = 2 memory = '2GB' queue = 'all.q' maxForks 3 errorStrategy 'ignore' input: file sample from raw_sample output: file("./sra2fq/${sample.baseName}") into map script: """ python2 $sra2fq -i ${sample} -f $fastq_dump -workpath $workpath -tag 'sra2fq ($sample.baseName)' """ }. after executing such process, I can see three jobs had been submitted. typing snodes in sge shell and i can see 6 cpu and 12G virtual memory has been used. I think CPU amount is properly but virtual memory is not. because every process occupy 2G memory so three such process such occupy 6G but not 12
arontommi
@arontommi
Sep 11 2018 11:16
i have some executables in the bin folder of my nf project, for some reason bin is not added to my $PATH, what did i break ?
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:21
Which Nextflow version are you using?
arontommi
@arontommi
Sep 11 2018 11:21
0.31.1
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:21
I'll try it as well
arontommi
@arontommi
Sep 11 2018 11:22
in the .command.run for the project it is clearly exported :
export PATH="/castor/project/proj_nobackup/b2017041_nobackup/RNAseq_AVC_test/RNAseq_VAX/bin:\$PATH"
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:24
Are your executables executable?
arontommi
@arontommi
Sep 11 2018 11:25
yes
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:25
What is your actual error?
Can you find a more detailed one in .nextflow.log?
arontommi
@arontommi
Sep 11 2018 11:27
Command error:
python3: can't open file 'HaplotypeCaller2tab.py': [Errno 2] No such file or directory
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:29
your script is not in bin, it's in bin/vcf2tab
I'm guesssing that vcf2tab is not in your $PATH
arontommi
@arontommi
Sep 11 2018 11:30
no i fixed that (not in most recent version on github)
Maxime Garcia
@MaxUlysse
Sep 11 2018 11:30
ok
that was a good guess
arontommi
@arontommi
Sep 11 2018 11:30
yea i was sure that was the problem when i discovered it
arontommi
@arontommi
Sep 11 2018 13:19
Yea i figured it out, it was just me being thick
Maxime Garcia
@MaxUlysse
Sep 11 2018 13:20
What was the issue then?
arontommi
@arontommi
Sep 11 2018 13:20
#!/usr/bin/env python not being included
woops
Maxime Garcia
@MaxUlysse
Sep 11 2018 13:20
LOL
arontommi
@arontommi
Sep 11 2018 13:30
reminder to myself, before blaming basically everything around me, read through my code a little slower.
Félix C. Morency
@fmorency
Sep 11 2018 19:54
@pditommaso Hey. Do you know if the S3 support built-in NF would work with on-premise S3 deployment tools such as minio?