These are chat archives for nextflow-io/nextflow

8th
Nov 2017
Paolo Di Tommaso
@pditommaso
Nov 08 2017 08:19
people interested in Bioconda/Biocontainers support, please comment nextflow-io/nextflow#493
Evan Floden
@evanfloden
Nov 08 2017 08:20
@fwip Thanks for spotting the typo!
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:23
Hi @pditommaso I'm trying out AWS for our pipeline, and I have trouble accessing some files in an s3 bucket when I'm using {} to specify multiple files, but not the other files, do you by any chance have any pointers to help me solve that?
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:24
oh, not yet released and already a bug ?! :grimacing:
please open an issue with an example and the error report
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:27
And not related at all, but we had a user that had trouble with our pipeline, and we stumble upon what might be a problem, we often find .command.trace file with only one number inside, I'm gathering more information before making a ticket too
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:28
are you using biocontainers ?
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:29
No
But I find the initiative interesting
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:30
I was mentioning in relation to the .command.trace problem
any container at all ?
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:30
sorry
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:30
no pb :)
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:30
Yes, we were using Singularity containers for this one, but I think I managed to reproduce with Docker containers
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:32
make sure the ps command in the container support state, pcpu and pmem options
eg ps -o pid= -o state= -o pcpu= -o pmem= -o vsz= -o rss=
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:32
I'll look more into it ;-) thanks for the pointer
Paolo Di Tommaso
@pditommaso
Nov 08 2017 09:33
also check that exists the pseudo file /proc/$$/io
Maxime Garcia
@MaxUlysse
Nov 08 2017 09:35
Got it
Francesco Strozzi
@fstrozzi
Nov 08 2017 11:26
@MaxUlysse I think we spotted the same bug :)
I’ve commented on your issue
Maxime Garcia
@MaxUlysse
Nov 08 2017 13:13
@fstrozzi I did not notice, I was still using the 0.25.7 and not the 0.26.0... sorry @pditommaso
Maxime Garcia
@MaxUlysse
Nov 08 2017 13:20
@pditommaso But now, nextflow is broke on the AMI ami-43f49030
Made an issue for that
Félix C. Morency
@fmorency
Nov 08 2017 14:46

Say I have a channel that emits

[100, A, B, foo, bar]
[100, B, A, bar, foo]
[100, A, C, foo, bar]
[100, C, A, bar, foo]
[100, C, B, foo, bar]
[100, B, C, bar, foo]

Is there any way to modify the channel to only keep

[100, A, B, foo, bar]
[100, A, C, foo, bar]
[100, B, C, bar, foo]

?

Félix C. Morency
@fmorency
Nov 08 2017 14:58
Mmm .choice() seems the way to go
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:34
@fmorency .map is the solution to half of NF problems .. :)
oops, didn't get right your question
maybe filter ?
Félix C. Morency
@fmorency
Nov 08 2017 16:35
I did it using choice
Works.
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:35
ok
Mike Smoot
@mes5k
Nov 08 2017 16:38
Hi @pditommaso just wanted to follow up on my comment about slurm yesterday. It seems as though there is a slurm bug. What seems to have happened is that my first task (4888) got submitted and assigned a job id, then slurmctld restarted to add nodes to the cluster, but because slurm somehow hadn't quite reified the task, it failed to serialize it properly, so when slurmctld restarted the task got purged. Then along comes my next task (4893) and it gets assigned the same job id and confusion reigns. This appears very closely related to this bug. I'm going to try upgrading slurm to see if that helps. Or maybe try AWS Batch!
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:39
I have no experience using SLURM in the cloud, but this confirms my idea that legacy batch schedulers are suited for cloud deployment
Mike Smoot
@mes5k
Nov 08 2017 16:42
Yeah, slurm's approach to scaling is pretty awful. Maybe I need to prioritize experimenting with AWS batch. So many projects, so little time! :)
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:42
don't tell me ! :)
Félix C. Morency
@fmorency
Nov 08 2017 16:48
@pditommaso submitting abstract today :sparkles:
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:48
hope there's a mention there! :)
Félix C. Morency
@fmorency
Nov 08 2017 16:49
Yeah you are in the refs
All processing was done using a Nextflow6 pipeline with all software dependencies bundled in a Singularity7 container ensuring quick and easy reproducibility of the results.
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:49
:clap:
what's the journal ?
Félix C. Morency
@fmorency
Nov 08 2017 16:49
I'll let you know the sponsor if I can
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:50
ah, it's a conference ?
Félix C. Morency
@fmorency
Nov 08 2017 16:51
yea
Paolo Di Tommaso
@pditommaso
Nov 08 2017 16:51
:+1:
Maxime Garcia
@MaxUlysse
Nov 08 2017 19:59
🍾