These are chat archives for nextflow-io/nextflow

14th
Jun 2018
Dave Istanto
@DaveIstanto
Jun 14 2018 01:00

hi guys, does anyone know how to use the bash substring method (like ${var#*/}) in nextflow shell? It seems that it's only finding unexpected char: # as follows:


N E X T F L O W  ~  version 0.30.1
Launching `testing.nf` [friendly_payne] - revision: a675915228
ERROR ~ unexpected char: '#' @ line 32, column 26.
       echo ${fileNameWithIndex#*+}
                            ^

1 error

thank you!

Radoslaw Suchecki
@bioinforad_twitter
Jun 14 2018 01:11
I haven't tried but you may be mixing up bash and nextflow variables. See docs: script and shell. Would help to see your process definition @ShawnConecone
so e.g. this might work \${var#*/}
Dave Istanto
@DaveIstanto
Jun 14 2018 02:12

thanks @bioinforad_twitter ,
here's the script:


fileChannel = Channel.from("SampleA+1", "SampleB+2")
fileIndexChannel = Channel.from(1,2)
process echoPairs {

        input:
        val fileName from fileChannel 
        val fileIndex from fileIndexChannel

        output:
        stdout into stdoutChannel

        shell:
        """ 
        echo ${fileName}
        """ 

}   

stdoutChannel.subscribe{ println it}

the objective here is to output "1" from "SampleA+1" and "2" from "SampleB+2"

currently it only outputs SampleA+1 and SampleB+2, and when # is added, or \${fileName#*/} , it's not outputting the right thing.

Radoslaw Suchecki
@bioinforad_twitter
Jun 14 2018 02:29
fileName is a NF variable, I suspect you are making it difficult for yourself by trying to stick to bash for string manipulation, but it is doable. e.g.
        script:
        """
        f=${fileName} #assign NF var value to bash var
        echo \${f#*+}
        """
Dave Istanto
@DaveIstanto
Jun 14 2018 03:54
hmm I see, Thank you very much @bioinforad_twitter
Anthony Underwood
@aunderwo
Jun 14 2018 07:16

Hi all. What's current best practice for running the nextflow master process for a long running set of jobs.
nohup nextflow run ......... &

??

Maxime Garcia
@MaxUlysse
Jun 14 2018 07:26
Personnaly I use a screen session
Maxime Vallée
@valleem
Jun 14 2018 07:26
@aunderwo there is a -bg flag
Anthony Underwood
@aunderwo
Jun 14 2018 07:28
@valleem - interesting will check that out. Do you use this?
Maxime Vallée
@valleem
Jun 14 2018 07:28
yes all the time
I would prefer in a screen but the HPC I use log me to different front nodes each time.
Maxime Garcia
@MaxUlysse
Jun 14 2018 07:29
@valleem can't you choose a specific node?
Maxime Vallée
@valleem
Jun 14 2018 07:30
I would like, yes, that is what I do on other HPC, but the main one, I cannot. It is obfuscated.
Anthony Underwood
@aunderwo
Jun 14 2018 07:31
when you run with -bg how do you check on progress?
does it write stdout to file
Maxime Garcia
@MaxUlysse
Jun 14 2018 07:31
why would they do that?
Maxime Vallée
@valleem
Jun 14 2018 07:31
@aunderwo no it just pops up when new jobs are submitted
Anthony Underwood
@aunderwo
Jun 14 2018 07:32
can't see documentation for bg
Maxime Vallée
@valleem
Jun 14 2018 07:32
@MaxUlysse it is understandable, they do not want people to hog up one login node
Maxime Garcia
@MaxUlysse
Jun 14 2018 07:33
Yes, but it's complicated for user to log to the same node each time
But I do see the point
Maxime Vallée
@valleem
Jun 14 2018 07:33
Anthony Underwood
@aunderwo
Jun 14 2018 07:36
@valleem Thanks - so it's basically just backgrounding the process. As you say screen would be nicer if your HPC env was more amenable!!
These guys recommend screen
http://andersenlab.org/dry-guide/quest-nextflow/
Maxime Vallée
@valleem
Jun 14 2018 07:38
@aunderwo yes exactly, nothing fancy (and it stores the pid as well). Yes screen is the best for this use.
Maxime Garcia
@MaxUlysse
Jun 14 2018 07:44
You can use tmux also if you're more familliar with it
Anthony Underwood
@aunderwo
Jun 14 2018 07:44
@MaxUlysse I've used both tmux and screen. tmux is definitely more powerful but not always installed
Maxime Garcia
@MaxUlysse
Jun 14 2018 07:49
Yes, but you can always try to request an install for it
Luca Cozzuto
@lucacozzuto
Jun 14 2018 08:20
I usually use -bg since I have the same problem of obfuscation
and I redirect the output to a log.txt file
then a simple tail -f log.txt indicates me what is going on
Anthony Underwood
@aunderwo
Jun 14 2018 08:23
@lucacozzuto Redirect using
> log.txt 2>&1
?
Luca Cozzuto
@lucacozzuto
Jun 14 2018 08:25
nextflow run mypipe.nf -bg > log.txt
nextflow sends everything to stdout
and more info are available in .nextflow.log
Paolo Di Tommaso
@pditommaso
Jun 14 2018 08:35
what problem of obfuscation ?
Lavanya Veeravalli
@veeravalli
Jun 14 2018 09:15
I have tried nextflow on sge as well as pbs scheduler. I get to see the native id in trace.txt for sge but I am not able to see that for pbs setup. Have anyone experience this before or am I missing something cat trace.txt as below name duration realtime peak_rss read_bytes write_bytes gatk_hc (region 33bfd2db3 for sample WHB508) 4m 39s 4m 33s 1.7 GB 0 2.8 MB
Luca Cozzuto
@lucacozzuto
Jun 14 2018 09:25
Hi obfuscated @pditommaso, is there a way to change the exit code in a process? I.e. I want to send a code to the system
Paolo Di Tommaso
@pditommaso
Jun 14 2018 09:27
exit 123
Luca Cozzuto
@lucacozzuto
Jun 14 2018 09:28
thanks
rfenouil
@rfenouil
Jun 14 2018 09:36
Hello there, sorry if the question has been answered already but I cannot find it in the documentation. Is there a unique option that causes nextflow to produce all its output (log, dag, report, workdir, '.nextflow'...) in a specific directory ?
So far I successfully redirected 'work' and 'temp' files using 'NXF_WORK' and 'NXF_TEMP' environment variables but I didn't find a way to do it for all files produced in 'current directory' when executing my workflow.
Maxime Vallée
@valleem
Jun 14 2018 09:37

@pditommaso

what problem of obfuscation ?

no worries, we were talking on the use of screen vs -bg because we sometimes face some troubles with screen when the login nodes are randomly assigned, and we cannot find our screen back.

Maxime Garcia
@MaxUlysse
Jun 14 2018 09:40
@rfenouil for log, dag and report you can specify where to save them in the config
report {
  enabled = true
  file = "/where/you/want/report.html"
}
same idea for the rest
rfenouil
@rfenouil
Jun 14 2018 09:41
Hi @MaxUlysse thank you. Indeed one can specify it individually but I wonder if there is a unique option to do it for all files produced
and specially, I did not find a way to specify the location of ".nextflow" folder
Maxime Garcia
@MaxUlysse
Jun 14 2018 09:42
that's a good question
Why do you want to do that?
rfenouil
@rfenouil
Jun 14 2018 09:43
Changing current directory before execution should be a solution but I would prefer to avoid that if possible
Maxime Garcia
@MaxUlysse
Jun 14 2018 09:44
In my case, I make one run in one particular directory
But I understand why you would want to launch everything just from one single place
rfenouil
@rfenouil
Jun 14 2018 09:46
@MaxUlysse I grouped all the 'scripts' (including nextflow ones) for a specific analysis in a directory, next to another folder with my data, and another one where the results go... All scripts are started from the script folder so I would like NF to write temp files and results etc in the specified result directory and not add anything into my script folder.
Maxime Garcia
@MaxUlysse
Jun 14 2018 09:51
I see, for Sarek we do have several scripts as well, so we made an environnement variable, and then we call everything using that variable in the destination directory
rfenouil
@rfenouil
Jun 14 2018 09:55

I see, thank you for the suggestion. However my analysis is very modest and far from being automated (some scripts to launch manually) and I would like to allow the user to start all commands sequentially while sitting in the script directory (and save him some time and headaches by not asking him to change directory).

In a similar taste a friend was trying to execute a NF script from a folder where she did not have write permission, a simple 'execdir' option to redirect all output would have solved this problem easily.

Apologies if that option exists already, I might have missed it while reading the doc.
lucky
@Lucky-lai
Jun 14 2018 10:46
Hi,i found the script in $baseDir/bin don't have permission to exec
it need to chmod -R $baseDir/bin every time?
Paolo Di Tommaso
@pditommaso
Jun 14 2018 10:51
Nope, git is supposed bro preserve file permissions
Sven F.
@sven1103
Jun 14 2018 11:11
@pditommaso the tests for #681 failed, because of API rate limit was hit:
+/home/travis/build/nextflow-io/nextflow/nextflow run nextflow-io/hello -name third --command hola
   API rate limit exceeded -- Provide your GitHub user name and password to get a higher rate limit
otherwise I am all set
Félix C. Morency
@fmorency
Jun 14 2018 13:58
:clap:
Bioninbo
@Bioninbo
Jun 14 2018 14:04
:clap:
Paolo Di Tommaso
@pditommaso
Jun 14 2018 14:08
feel free to propose or add new patterns
Mike Smoot
@mes5k
Jun 14 2018 17:30
@pditommaso The patterns are excellent! Now trying to wrap my head around the feedback loop... :)
Félix C. Morency
@fmorency
Jun 14 2018 17:50
@mes5k Woah. Didn't know one could do that in NF. Quite cool actually.