These are chat archives for nextflow-io/nextflow

26th
Jun 2018
LukeGoodsell
@LukeGoodsell
Jun 26 2018 09:52 UTC
Is there an easy way to assert that a channel has at least one item?
Maxime Garcia
@MaxUlysse
Jun 26 2018 09:54 UTC
Check if not empty ?
LukeGoodsell
@LukeGoodsell
Jun 26 2018 10:02 UTC
Great, thanks @MaxUlysse . So I’d need to do something like:
itemChannel = itemChannel
        .ifEmpty("blah")
        .map { item -> assert item != "blah"; item }
Actually, I see ifEmpty can call a function, so:
itemChannel = itemChannel
        .ifEmpty { assert 1 == 2 }
Thanks!
Maxime Garcia
@MaxUlysse
Jun 26 2018 11:19 UTC
You're welcome
Mike Smoot
@mes5k
Jun 26 2018 19:33 UTC
Hi @pditommaso I've got s3 file urls that look like s3://super-secret/290419642943253529 which I'm successfully able to consume in my pipeline, but the filename ends up being 290419642943253529. I'd like to be able to refer to it with a more human name (e.g. 'something.json.gz'). renameTo seems to want to rename the original file, whereas I just want nextflow to see the file as 'something.json.gz' and not rename the source. Can you think of an elegant way to do this? I know I can just mv 290419642943253529 something.json.gz, but I'd prefer to avoid that if possible.
I'm hoping to do this in a channel so that the name propagates everywhere the channel is used.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:45 UTC
not sure to understand, s3://super-secret/290419642943253529 is what? an input file ?
Mike Smoot
@mes5k
Jun 26 2018 19:47 UTC
Yes, e.g. Channel.fromPath("s3://super-secret/290419642943253529")
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:47 UTC
and you need to renamed in the context of the task execution ?
Mike Smoot
@mes5k
Jun 26 2018 19:48 UTC
not so much need as want.
but yes, in the task execution context
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:48 UTC
what about
input: 
file 'something.json.gz' from Channel.fromPath("s3://super-secret/290419642943253529")
Mike Smoot
@mes5k
Jun 26 2018 19:49 UTC
Ah, it's the quotes that are magic!
I was just doing
input:
file(x) from ...
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:50 UTC
hey, this is nextflow 1-0-1 :smile:
Mike Smoot
@mes5k
Jun 26 2018 19:52 UTC
my brain just doesn't have room for so many useful details. Sigh. :)
Michael L Heuer
@heuermh
Jun 26 2018 20:53 UTC
@pditommaso Please let me know when you're teaching that class. Hopefully I can find money to travel to Barcelona to attend. :smile:
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:10 UTC
@heuermh hey man, are you at BOSC already ?
Michael L Heuer
@heuermh
Jun 26 2018 21:19 UTC
I'm at the bar already
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:19 UTC
cool, see you in a while
Michael L Heuer
@heuermh
Jun 26 2018 21:19 UTC
Looking forward to it!
We might need to talk about Kubernetes at some point
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:21 UTC
looking forward about that, do you have specific needs ?
Michael L Heuer
@heuermh
Jun 26 2018 21:22 UTC
Supposedly Spark 2.3.x has built-in support for Kubernetes, perhaps that kills two birds with one stone
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:23 UTC
the second bird is supposed to be NF + Adam + Spark I guess :smile:
Michael L Heuer
@heuermh
Jun 26 2018 21:24 UTC
Yeah something like that :smile:
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:24 UTC
happy to chat about as we meet
Mike Smoot
@mes5k
Jun 26 2018 21:36 UTC
Hi @pditommaso I'm currently building nextflow from source and when I run ./launch.sh I'm getting Error: Could not find or load main class nextflow.cli.Launcher. Do I need to export a class path or something?
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:37 UTC
make to run clean ie. make clean compile
Mike Smoot
@mes5k
Jun 26 2018 21:37 UTC
ah, yes, I probably had some cruft lying around
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:38 UTC
likely, the launcher mess up the JARs with different version numbers
Mike Smoot
@mes5k
Jun 26 2018 21:40 UTC
yup, that works! I'm trying to debug an AWS authentication issue. We use temporary session tokens on our local machines to authenticate with AWS and I'm trying to get that working within nextflow. If I get something working I'll submit a patch.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:42 UTC
managing all possible AWS auth is an headache, it's a kind of temporary AWS keys ?
Mike Smoot
@mes5k
Jun 26 2018 21:44 UTC
yes, they're temporary keys that only last an hour. a HUGE headache
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:45 UTC
have you taken in consideration IAM Roles ?
the blue box
Mike Smoot
@mes5k
Jun 26 2018 21:47 UTC
Yes, and we're using an IAM role when running in AWS and that works great. This is just for running locally, which we still do a quite a bit of.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:48 UTC
I think it should be possible to use an instance role also for local auth
Mike Smoot
@mes5k
Jun 26 2018 21:50 UTC
I think so too, but our security people don't want us to.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:58 UTC
:+1: