These are chat archives for nextflow-io/nextflow

26th
Jun 2018
LukeGoodsell
@LukeGoodsell
Jun 26 2018 09:52
Is there an easy way to assert that a channel has at least one item?
Maxime Garcia
@MaxUlysse
Jun 26 2018 09:54
Check if not empty ?
LukeGoodsell
@LukeGoodsell
Jun 26 2018 10:02
Great, thanks @MaxUlysse . So I’d need to do something like:
itemChannel = itemChannel
        .ifEmpty("blah")
        .map { item -> assert item != "blah"; item }
Actually, I see ifEmpty can call a function, so:
itemChannel = itemChannel
        .ifEmpty { assert 1 == 2 }
Thanks!
Maxime Garcia
@MaxUlysse
Jun 26 2018 11:19
You're welcome
Mike Smoot
@mes5k
Jun 26 2018 19:33
Hi @pditommaso I've got s3 file urls that look like s3://super-secret/290419642943253529 which I'm successfully able to consume in my pipeline, but the filename ends up being 290419642943253529. I'd like to be able to refer to it with a more human name (e.g. 'something.json.gz'). renameTo seems to want to rename the original file, whereas I just want nextflow to see the file as 'something.json.gz' and not rename the source. Can you think of an elegant way to do this? I know I can just mv 290419642943253529 something.json.gz, but I'd prefer to avoid that if possible.
I'm hoping to do this in a channel so that the name propagates everywhere the channel is used.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:45
not sure to understand, s3://super-secret/290419642943253529 is what? an input file ?
Mike Smoot
@mes5k
Jun 26 2018 19:47
Yes, e.g. Channel.fromPath("s3://super-secret/290419642943253529")
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:47
and you need to renamed in the context of the task execution ?
Mike Smoot
@mes5k
Jun 26 2018 19:48
not so much need as want.
but yes, in the task execution context
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:48
what about
input: 
file 'something.json.gz' from Channel.fromPath("s3://super-secret/290419642943253529")
Mike Smoot
@mes5k
Jun 26 2018 19:49
Ah, it's the quotes that are magic!
I was just doing
input:
file(x) from ...
Paolo Di Tommaso
@pditommaso
Jun 26 2018 19:50
hey, this is nextflow 1-0-1 :smile:
Mike Smoot
@mes5k
Jun 26 2018 19:52
my brain just doesn't have room for so many useful details. Sigh. :)
Michael L Heuer
@heuermh
Jun 26 2018 20:53
@pditommaso Please let me know when you're teaching that class. Hopefully I can find money to travel to Barcelona to attend. :smile:
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:10
@heuermh hey man, are you at BOSC already ?
Michael L Heuer
@heuermh
Jun 26 2018 21:19
I'm at the bar already
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:19
cool, see you in a while
Michael L Heuer
@heuermh
Jun 26 2018 21:19
Looking forward to it!
We might need to talk about Kubernetes at some point
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:21
looking forward about that, do you have specific needs ?
Michael L Heuer
@heuermh
Jun 26 2018 21:22
Supposedly Spark 2.3.x has built-in support for Kubernetes, perhaps that kills two birds with one stone
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:23
the second bird is supposed to be NF + Adam + Spark I guess :smile:
Michael L Heuer
@heuermh
Jun 26 2018 21:24
Yeah something like that :smile:
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:24
happy to chat about as we meet
Mike Smoot
@mes5k
Jun 26 2018 21:36
Hi @pditommaso I'm currently building nextflow from source and when I run ./launch.sh I'm getting Error: Could not find or load main class nextflow.cli.Launcher. Do I need to export a class path or something?
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:37
make to run clean ie. make clean compile
Mike Smoot
@mes5k
Jun 26 2018 21:37
ah, yes, I probably had some cruft lying around
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:38
likely, the launcher mess up the JARs with different version numbers
Mike Smoot
@mes5k
Jun 26 2018 21:40
yup, that works! I'm trying to debug an AWS authentication issue. We use temporary session tokens on our local machines to authenticate with AWS and I'm trying to get that working within nextflow. If I get something working I'll submit a patch.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:42
managing all possible AWS auth is an headache, it's a kind of temporary AWS keys ?
Mike Smoot
@mes5k
Jun 26 2018 21:44
yes, they're temporary keys that only last an hour. a HUGE headache
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:45
have you taken in consideration IAM Roles ?
the blue box
Mike Smoot
@mes5k
Jun 26 2018 21:47
Yes, and we're using an IAM role when running in AWS and that works great. This is just for running locally, which we still do a quite a bit of.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:48
I think it should be possible to use an instance role also for local auth
Mike Smoot
@mes5k
Jun 26 2018 21:50
I think so too, but our security people don't want us to.
Paolo Di Tommaso
@pditommaso
Jun 26 2018 21:58
:+1: