+1
Bonjourno. Is there a way to have Nextflow abort in the event of a warning à la
options(warn = 2) in R, or use strictin Perl?
Buongiorno ! :)
nope, warn = 2 in NF is supposed to be an error, hence it will abort ..
what warning are you referring ?
I got a "WARN: Input tuple does not match input set cardinality declared by process …”
Fortunately, I spotted it and fixed a bug that could have broken my pipeline. I was hoping there may be a wat to have it fail at the warning so that potential bugs can be more easily spotted.
Fortunately, I spotted it and fixed a bug that could have broken my pipeline. I was hoping there may be a wat to have it fail at the warning so that potential bugs can be more easily spotted.
It wasn’t a bug in a
script/shell section but a change to the set(…) spec for a channel
(NB: the bug was mine, not Nextflow’s)
yes, it could have sense to have
strict flag at some point
Shall I create a feature request issue on GitHub?
yes
grazie
:)
de nada
(time to learn some italian in this channel)
:D
I agree
haha, I support this idea. I’m bored of English!
I use too much english at work
I don;t even know many technical words in italian
Like "Sequenziamento di ultima generazione"?
Segnalato: #426
better to keep italian for food ;)
I wonder if some bioinformatics groups / labs have already made a decision to not buy a large cluster but just to use AWS or similar.
labs tend to use internal infra
but there are companies/startups (also in this channel) just using the cloud
@Shellfishgene yes :)
that’s why I started discussing AWS Batch with Paolo
Does it come out cheaper for you, or is it also about space/administration?
it’s cheaper because we literally pay just what we use (I know, seems like advertising but it’s true) + a lot simpler administration, although you need to invest some time in learning how AWS works (especially security and users / permissions).
@Shellfishgene, I work at a startup and use AWS. There’s much lower up-front costs, and it allows us to prototype and test different infrastructures.
Our needs are constantly changing, and if we bought our own infrastructure, it would almost certainly have both excess capacity in some regards, and insufficient capacity in others.
That makes sense. I guess larger clusters at Universities are shared between enough people that they don't really run idle either.
yes exactly, for small groups or companies I think it starts to make much more sense to go to the cloud (AWS, Google, Azure…)
I have no data, but I imagine the economies of scale may make it cheaper for universities to use IAAS providers when their current clusters reach end-of-life. The data protection, flexibility and control considerations may still outweigh cost when it comes to the decision, though.
I think the typical uni cluster is added on to and ends up as a mishmash of nodes... Ours is actually leased it seems, and gets replaced every so often.
Plus I presume it may it easier, at our uni to use a cluster one has to send some sort of proposal and get it approved, while to use AWS one only has to pay :money_with_wings:
I am totally in favour of the cloud, but it involves anyway a certain effort for both the administrators and the users to adapt or change the way of doing certain things. Running workloads on EC2 is similar to what you do locally but there may be differences in the way users access data for example (S3 mainly, EFS is way too expensive, I think only used in few places). Using managed services like Batch or ECS to run containers imply also a shift in the approach for the users (and the administrators as well). What I mean is that for many universities the whole process could take months (and money) to train people to use a new platform.
What I have seen in some places is the hybrid approach, keep the local infra and attach the cloud when there is a peek in usage or special needs. This way for the users is more transparent and the administrators get the time to learn
mobility is the key, develop on the laptop, deploy in the uni cluster, scale in the cloud when needed
@pditommaso I just ran
nextflow self-update using sudo, as I have it in /opt/. It appears that also changes stuff in ~/.nextflow, for example the framkework dir. This is then owned by root and nextflow console does not work anymore. Easy to fix, but is it supposed to work like that?
NF is supposed to be installed without sudo
Ok, so system-wide installs are discouraged?
in a cluster environment ?
No, this is a on a single multi-user workstation. I guess it's so easy to install every user can do it.
I suggest to only share the
nextflow launcher script in /usr/local/bin
then it will pull the deps as needed
the drawback is that the self-update won't be able to overwrite the launcher because it's owned by root
We played with this kind of thing for a while, but now I just tell people to install their own user version
Seems easier and tends to work better we find
Totally unrelated question: is the use of
storeDir really discouraged?
unless you need to preserve data across different pipelines/instances
i would like to preserve the mask I computed, so that it awlays stays the same when I change other parameters
but somehow on monday storeDir did not work properly
and the process ran again, invalidating the cache of all the subsequent proceeses
Can anybody have touched those files?
That could be, because they were on our server
which is still in trouble
thats a good point
does it imply that the hash is computed on the base of the date and not of the file contents?
NF creates the task hashes by the file name and last modified timestap for each input, by default
if you want to use the file content use
cache 'deep'
:cool: Next feature request: custom hashing per task
:clap:
Amazing. Just tell us what nextflow does not do!!
:)
Now I found out where the problem was :smile: stupid me: the mask was correctly stored, but it depended on an upstream process whose input I changed. Now storing the upstream too, should take care of it
:+1:
@pditommaso would it make sense to modify the
buffer operator so that the opening and closing criteria could access to the buffers content as well?
the use case would be the following:
- I have a channel emitting sets consisting of files and dates.
- Whenever the time difference from starting to ending dates in the buffer is smaller than a certain threshold, the buffer contents should be emitter
To do so, I need a memory where I keep track of the starting date. At the moment I use an external variable that is used in the closure, but I would be somehow more elegant to directly provide it at the closure invokation time
Hi, I try to have this command:
'''awk '{if($2 == 4){ print "@" $1 "\n" $10 "\n+\n" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq'''
but it ends up as: awk '{if($2 == 4){ print "@" $1 "
" $10 "
+
" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq
" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq
^ unterminated string
etc ...
can you use proper formatting? it's hard to read
Nextflow turns the 3. " into a string terminator.
Oh there is markdown!
#!/bin/bash -ue
awk '{if($2 == 4){ print "@" $1 "
" $10 "
+
" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq
can you post the script section of your process?
or the entire process
#!/bin/bash -ue
awk '{if($2 == 4){ print "@" $1 "
" $10 "
+
" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq
in .command.sh
the process is just:
process extrac_unaligned_rrna {
input:
file "reads.umi.cut.rrna.sam" from rrna
output:
file "reads.umi.clean.fq" into clean
shell:
'''awk '{if($2 == 4){ print "@" $1 "\n" $10 "\n+\n" $11 }}' reads.umi.cut.rrna.sam > reads.umi.clean.fq'''
}Command error:
awk: cmd. line:1: {if($2 == 4){ print "@" $1 "
awk: cmd. line:1: ^ unterminated string
awk: cmd. line:1: {if($2 == 4){ print "@" $1 "
awk: cmd. line:1: ^ syntax errorI don't use shell: blocs, but from the Nextflow documentation
https://www.nextflow.io/docs/latest/process.html#shell
there is a note about
Shell script definition requires the use of single-quote ' delimited strings. When using double-quote " delimited strings, dollar variables are interpreted as Nextflow variables as usual. See String interpolation.
Did you try escaping the "\n" such as "\\n"?
yes indeed I had to escape the \n
Thank you very much @fmorency
You're welcome.
Is there a way to prevent nf from using the 'local' executor even when there is no config file?
@Shellfishgene I guess you could pass it on the command line such as
nextflow run -process.executor slurm ...
I meant more as global config setting somewhere, so I don't accidentally run jobs on the head node of the cluster when I forget to specify or copy the config file...
@Shellfishgene You can define the environment variable
NXF_EXECUTOR with the default executor you want to use
Cool, thanks!