These are chat archives for nextflow-io/nextflow
nextflow -c <config file> cloud
nextflow cloud create, -c is used to specify the number of instances. I guess positional arguments matter
keyHashin place of that
main.nfinstead of letting nextflow download the whole thing ?
Hi all, I was wondering if anyone had experimented with the AWS x1.32xlarge instances? I'm seeing a weird Nextflow error that I hadn't seen with other (smaller) instance types. During an initial pipeline step, when NF is downloading data from S3, I'm seeing "Timeout waiting for connection from pool" errors from the underlying Java SDK.
Is there anything in the S3 implementation in NF that scales with the number of available processors? I was wondering if with 128 processors, it might be hitting limits that it wouldn't have with fewer processors. If someone could point me towards the relevant code I can go exploring myself.
.nextflow.logor at least the complete stack trace ?
CAPSULE: Downloading dependency io.nextflow:nxf-httpfs:pom:0.22.6 CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.nextflow:nxf-httpfs:pom:0.22.6 in central (https://repo1.maven.org/maven2/) (for stack trace, run with -Dcapsule.log=verbose)
NXF_VER=0.22.5 nextflow run .. etc
OK, it's consistent, at least -- it's failed every time, and works fine on a smaller instance. It looks like ~60 file copy requests were executing in parallel when it died.
Dec-12 23:05:06.538 [main] DEBUG nextflow.cli.CmdRun - Version: 0.22.6 build 4117 Modified: 12-12-2016 22:04 UTC (23:04 CEST) System: Mac OS X 10.11.5 Runtime: Groovy 2.4.7 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15 Encoding: UTF-8 (UTF-8) Process: firstname.lastname@example.org [192.168.1.46] CPUs: 8 - Mem: 16 GB (2 GB) - Swap: 1 GB (897 MB)
Dec-12 21:34:01.398 [main] DEBUG nextflow.cli.CmdRun - Version: 0.22.6 build 4116 Modified: 04-12-2016 21:50 UTC System: Linux 3.19.0-74-generic Runtime: Groovy 2.4.7 on OpenJDK 64-Bit Server VM 1.7.0_121-b00 Encoding: UTF-8 (UTF-8) Process: 36056@ip-10-101-9-244 [10.101.9.244] CPUs: 128 - Mem: 1.9 TB (1.9 TB) - Swap: 0 (0)
NXF_VER=0.23.0-SNAPSHOT nextflow run .. etc