These are chat archives for nextflow-io/nextflow

9th
Mar 2019
tbugfinder
@tbugfinder
Mar 09 00:11
Which container are you using
Olga Botvinnik
@olgabot
Mar 09 22:02
@tbugfinder From nextflow.config:
process.executor = 'awsbatch'
process.queue = 'nextflow'
process.container = 'ubuntu'

executor.awscli = '/home/ec2-user/miniconda/bin/aws'

aws {
    region = 'us-west-2'

    client {
        maxConnections = 20
        connectionTimeout = 10000
        uploadStorageClass = 'INTELLIGENT_TIERING'
        storageEncryption = 'AES256'
    }
}
I'm also having trouble with remote vs local. On my laptop, I can run this workflow on AWS batch and get this error, but on a remote machine, I'm getting a 403 access denied even though AWS configure is set up properly:
 Fri  8 Mar - 13:40  ~/code/nf-kmer-similarity   origin ☊ master ✔ 14☀ 
  make run_aws
sudo nextflow run main.nf \
        -work-dir s3://olgabot-maca/nextflow-workdir-test/ \
        -bucket-dir s3://olgabot-maca/nextflow-bucket-dir-test/ \
        -with-trace -with-timeline -with-dag -with-report -latest -resume
N E X T F L O W  ~  version 19.01.0
Launching `main.nf` [nasty_montalcini] - revision: 3c5e3e4bc9
ERROR ~ Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2A78AFC8ADB32CFD; S3 Extended Request ID: npGZNxWvBGlGYx1+JNOAen1Btgm1uf9srwKkI6XExGT5X9spHKSg2ReFppAyEsLhHbpFYCHFNFU=)

 -- Check '.nextflow.log' file for details
Makefile:2: recipe for target 'run_aws' failed
make: *** [run_aws] Error 1

 ✘  Sat  9 Mar - 13:54  ~/code/nf-kmer-similarity   origin ☊ master ✔ 14☀ 
  aws configure
AWS Access Key ID [****************N4FQ]:
AWS Secret Access Key [****************mAog]:
Default region name [us-west-2]:
Default output format [None]:
Olga Botvinnik
@olgabot
Mar 09 22:08
And here's my local, laptop config:
(base)
 ✘  Sat  9 Mar - 14:06  ~/code/nf-kmer-similarity   origin ☊ master 42☀ 2● 
  hostname
Olgas-MacBook-Pro.local
(base)
 Sat  9 Mar - 14:07  ~/code/nf-kmer-similarity   origin ☊ master 42☀ 2● 
  aws configure
AWS Access Key ID [****************N4FQ]:
AWS Secret Access Key [****************mAog]:
Default region name [us-west-2]:
Default output format [json]:
Ah I misremembered, this configure doesn't work on my local, either! Huzzah!
I'm confused because I ran the tutorial just fine:
(base)
 ✘  Fri  8 Mar - 12:25  ~/code/nextflow-test   origin ☊ master 10432‒ 
  make
nextflow run tutorial.nf \
        -work-dir s3://olgabot-maca/nextflow-workdir-test/ \
        -bucket-dir s3://olgabot-maca/nextflow-bucket-dir-test/ \
        -with-trace -with-timeline -with-dag -with-report -latest -resume
N E X T F L O W  ~  version 19.01.0
Launching `tutorial.nf` [focused_yalow] - revision: 361b274147
[warm up] executor > awsbatch
[1e/f709a8] Submitted process > splitLetters
[e1/fb6083] Submitted process > convertToUpper (1)
[cd/279a72] Submitted process > convertToUpper (2)
HELLO
WORLD!
(base)
Olga Botvinnik
@olgabot
Mar 09 22:19
Here's the compute environment:
***** Overview
****** Compute environment name nextflow-v3-from-blog
***** Compute environment ARN arn:aws:batch:us-west-2:423543210473:compute-environment/nextflow-v3-from-blog
***** Type MANAGED
***** Status VALID
***** State ENABLED
***** Service role arn:aws:iam::423543210473:role/AWSBatchServiceRole
****** AWSBatchServiceRole
****** AmazonEC2SpotFleetTaggingRole
***** Compute resources
****** Minimum vCPUs 0
****** Desired vCPUs 0
****** Maximum vCPUs 2048
****** Instance types optimal
****** Launch template --
****** Launch template version
***** Instance role arn:aws:iam::423543210473:instance-profile/nextflow-ECSInstanceRole
****** AmazonS3FullAccess
****** AmazonEC2ContainerServiceforEC2Role
***** Spot fleet role arn:aws:iam::423543210473:role/aws-ec2-spot-fleet-role
****** AmazonS3FullAccess
****** AmazonEC2SpotFleetAutoscaleRole
****** AmazonEC2SpotFleetRole
****** AmazonEC2SpotFleetTaggingRole
***** EC2 Keypair
***** AMI idami-0c323ba3e98b979f9
***** vpcId vpc-7b0f7d1c
***** Subnets subnet-86d562ad, subnet-672e832e, subnet-04119a63, subnet-4347451b
***** Security groups sg-3195a049
***** Placement group
Adding AmazonS3FullAccess to AWSBatchServiceRole doesn't fix it either
As a user, I have:
  • AdministratorAccess
  • AmazonEC2FullAccess
  • AmazonS3FullAccess
  • AWSBatchFullAccess
Olga Botvinnik
@olgabot
Mar 09 23:09
Turns out the permissions issue was because I was running the command as root, and root doesn't have AWS credentials -_-;;; :sob:
Michael L Heuer
@heuermh
Mar 09 23:12
@drdozer It's been a while! There will be some NF folks at BOSC and BOSC Codefest