These are chat archives for nextflow-io/nextflow

16th
Sep 2017
Anthony Underwood
@aunderwo
Sep 16 2017 10:29
Hi - if running Nextflow on AWS how are local input files transferred to the cloud? Do they need to have been uploaded to EFS first?
Phil Ewels
@ewels
Sep 16 2017 10:34
EFS / s3 / FTP or anywhere accessible to the cloud, yup
Anthony Underwood
@aunderwo
Sep 16 2017 10:54
Thanks @ewels how do I then specify the path to the file?
in s3 using s3://my-bucket/data/sequences.fa
ftp via ftp://....
in EFS ???
Mike Smoot
@mes5k
Sep 16 2017 13:30
EFS is just a normal file path on the EFS filesystem (aka Amazon's version of NFS). So something like /mnt/efs/whatever/sequences.fa assuming you've mounted the EFS drive on /mnt/efs
Anthony Underwood
@aunderwo
Sep 16 2017 13:32

Hi when I'm trying to use AWS with

nextflow cloud create my-cluster -c 2

it creates a cluster fine and exits with

Login in the master node using the following command:
  ssh -i /Users/anthony/.ssh/id_rsa anthony@ec2-54-194-64-221.eu-west-1.compute.amazonaws.com

However it never connects - port 22 seems to be inaccessible

ssh -v -i /Users/anthony/.ssh/id_rsa anthony@ec2-54-194-64-221.eu-west-1.compute.amazonaws.com
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to ec2-54-194-64-221.eu-west-1.compute.amazonaws.com [54.194.64.221] port 22.
I can create an instance via AWS console and connect fine via ssh so there's no network issue
Mike Smoot
@mes5k
Sep 16 2017 13:37
Is the user anthony or should it be ec2-user?
And is /Users/anthony/.ssh/id_rsa the key you use for AWS?
Anthony Underwood
@aunderwo
Sep 16 2017 13:45
nextflow says I should use anthony I think:
> Launch configuration:
 - driver: 'aws'
 - imageId: 'ami-43f49030'
 - instanceType: 'm4.xlarge'
 - keyFile: /Users/anthony/.ssh/id_rsa.pub
 - spotPrice: 0.06
 - subnetId: 'subnet-b2af26d7'
 - userName: 'anthony'
Mike Smoot
@mes5k
Sep 16 2017 13:50
I haven't played with nextflow cloud in a while so I'm not sure. I'd guess there's still an ec2-user which I would try. Can you see the instance in your AWS console? Usually clicking around there will give you pointers on how to ssh in.
Anthony Underwood
@aunderwo
Sep 16 2017 13:51
yes I can see the instance in EW2. However it's not even connecting to check authentication :(
Anthony Underwood
@aunderwo
Sep 16 2017 14:02

When I try to connect to my t2.nano instance with telnet I get:

telnet 54.229.124.201 22
Trying 54.229.124.201...
Connected to ec2-54-229-124-201.eu-west-1.compute.amazonaws.com.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2

However when I try to connect to the nextflow instance:

telnet 34.253.204.214 22
Trying 34.253.204.214...

Port 22 is not open

Mike Smoot
@mes5k
Sep 16 2017 14:05
Is that the right IP address?
Anthony Underwood
@aunderwo
Sep 16 2017 14:06
yes copied from the EC2 console :)
It was a new cluster - so differs from the IP from a few posts ago. Not point having cluster running if I can't connect!!
Mike Smoot
@mes5k
Sep 16 2017 14:07
Ah
Anthony Underwood
@aunderwo
Sep 16 2017 14:07
The ascii cast on the Nextflow blog by @pditommaso seemed so simple so I must be missing something!
Mike Smoot
@mes5k
Sep 16 2017 14:09
Are you using a custom AMI or a nextflow one?
Anthony Underwood
@aunderwo
Sep 16 2017 14:10
the nextflow one
cloud {
    imageId = 'ami-43f49030'
    instanceType = 'm4.xlarge'
    spotPrice = 0.06
    subnetId = 'subnet-b2af26d7'
}
Mike Smoot
@mes5k
Sep 16 2017 14:11
good
Anthony Underwood
@aunderwo
Sep 16 2017 16:25

An nmap of my instance launched from the console reveals that 22 is open

Starting Nmap 6.47 ( http://nmap.org ) at 2017-09-16 17:21 BST
Nmap scan report for ec2-54-194-203-142.eu-west-1.compute.amazonaws.com (54.194.203.142)
Host is up (0.060s latency).
Not shown: 997 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
554/tcp  open  rtsp
7070/tcp open  realserver

Same for the master node started by nextflow reveals that 22 is not open

Starting Nmap 6.47 ( http://nmap.org ) at 2017-09-16 17:24 BST
Nmap scan report for ec2-54-229-109-184.eu-west-1.compute.amazonaws.com (54.229.109.184)
Host is up (0.025s latency).
Not shown: 998 filtered ports
PORT     STATE SERVICE
554/tcp  open  rtsp
7070/tcp open  realserver
Paolo Di Tommaso
@pditommaso
Sep 16 2017 16:57
I'm a bit lost with this, what's the problem you are experiencing ?
Anthony Underwood
@aunderwo
Sep 16 2017 16:58
if I run nextflow cloud create my-cluster -c 1
it tells me that the cluster is up but then when I follow the instructions to connect to the master node the ssh command hangs
port 22 is not open on the server as proven by nmap command
Paolo Di Tommaso
@pditommaso
Sep 16 2017 16:59
that's bad, you ssh to login
Anthony Underwood
@aunderwo
Sep 16 2017 16:59
Also telnet to the server on port 22 never connects
Paolo Di Tommaso
@pditommaso
Sep 16 2017 16:59
could it be a problem with your security group ?
Anthony Underwood
@aunderwo
Sep 16 2017 17:00
@pditommaso yes via ssh
If i make an instance via AWS console. All is good
do I need to specify a security group to nextflow?
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:01
let me check
Anthony Underwood
@aunderwo
Sep 16 2017 17:02
the launch wizard on AWS console creates security groups that presumably allow 22
maybe I need to add 22 to the default security group?
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:02
that's surely would be :)
I don't remember if the subnet is linked somehow to a security group
otherwise you need to make sure the default one opens the require ports or specify one in the NF config
Anthony Underwood
@aunderwo
Sep 16 2017 17:04
image.png
does this look sensible?
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:06
add also a rule to open all in-bound connection in the same security group both for TCP and UDP
ie source the security group itself
Anthony Underwood
@aunderwo
Sep 16 2017 17:10
Do you think it would be helpful to have an option to specify a security group from Nextflow?
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:11
there's !
Anthony Underwood
@aunderwo
Sep 16 2017 17:12
Oh wow cool
Hey guess what I'm in now that I've set a security group - I knew it would be something obvious!
Thanks
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:14
:+1:
Anthony Underwood
@aunderwo
Sep 16 2017 17:15
Followed the workshop on twitter. Looked incredible!
Sorry I couldn't make it but I'm sure my colleague Steve learnt loads!
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:16
Yes, we are all very happy !
Anthony Underwood
@aunderwo
Sep 16 2017 17:16
I am starting a new job soon and will be trying to persuade them to use Nextflow!
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:16
there was a lot of interest and feedback, super useful!
cool !
Anthony Underwood
@aunderwo
Sep 16 2017 17:17
I see from Twitter that modularisation was popular. Certainly I'd find it useful to have code broken up when the workflows start to get super long
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:18
yes, everybody wants that and we are going into it
Anthony Underwood
@aunderwo
Sep 16 2017 17:19
:clap:
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:19
:+1:
Anthony Underwood
@aunderwo
Sep 16 2017 17:20
It's been an eye opener for my group and for a group I'm supervising in Oxford. It's bringing the 2 groups together. Thanks for an amazing product!
Anthony Underwood
@aunderwo
Sep 16 2017 17:49
When I login into the head node and try and run nextflow it says nextflow can not be found
ahhh - needed to do ./nextflow
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:50
yes, it's not un the path
maybe we should improve this
Anthony Underwood
@aunderwo
Sep 16 2017 17:51
no worries

It's now saying there are no bams in my S3 path?

s3://nextflow-data/mykrobe-bams/*.bam

Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:51
it's a question or an assertion ?
Anthony Underwood
@aunderwo
Sep 16 2017 17:52
Does the S3 bucket need to be in the same region? It's saying
Exception in thread "Thread-3" com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: null.Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: 33477E964E60F15D), S3 Extended Request ID: 78B69FwXRcyNe5N8Rd2PGXGyEpsMvSg3PZCUXdwkTeosgmGmBtnnycaOfVUbMu8PkqF9zLxNWuA=
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:53
I don't think so ..
maybe a security permission problem on that bucket ?
are you able to access it with aws s3 ls .. ?
Anthony Underwood
@aunderwo
Sep 16 2017 17:54
OK I'll take a look
sorry got to sign off now- Speak later. Thanks for your help as always :)
Paolo Di Tommaso
@pditommaso
Sep 16 2017 17:56
no pb
it's supposed to be saturday night ;)