These are chat archives for dereneaton/ipyrad

9th
Jun 2018
jianlizhao
@jianlizhao
Jun 09 2018 01:01

@isaacovercast Hi Isaac, thank you. here are the complete message:ipyrad [v.0.7.24]

Interactive assembly and analysis of RAD-seq data

loading Assembly: lowdepth6step4
from saved path: ~/Documents/GBS2/Rtibetica_others/lowdepth6step4.json
establishing parallel connection:
host compute node: [40 cores] on service01

Step 7: Filter and write output files for 223 Samples
[####################] 100% filtering loci | 0:06:14
ERROR:ipyrad.assemble.write_outfiles:error in filter_stacks on chunk 36894: EngineError(Engine 'e74af511-46d5b371405e41849614a7af' died while running task u'75750931-76ea748c6a819abcbacee04d')
ERROR:ipyrad.core.assembly:IPyradWarningExit: error in filter_stacks on chunk 36894: EngineError(Engine 'e74af511-46d5b371405e41849614a7af' died while running task u'75750931-76ea748c6a819abcbacee04d')

Encountered an error (see details in ./ipyrad_log.txt)
Error summary is below -------------------------------
error in filter_stacks on chunk 36894: EngineError(Engine 'e74af511-46d5b371405e41849614a7af' died while running task u'75750931-76ea748c6a819abcbacee04d')
warning: error during shutdown:
[Errno 3] No such process
ERROR:ipyrad.core.assembly:shutdown warning: [Errno 3] No such process

Isaac Overcast
@isaacovercast
Jun 09 2018 02:08
@jianlizhao Did you verify your resource allocation? we recommend 4GB per core. EngineErrors are almost always resource depletion.
jianlizhao
@jianlizhao
Jun 09 2018 12:23
@isaacovercast I restarted the server and used 10 cores. It worked. But the "writting vcf file" is running more than five hours.Should I stop it or waiting?:Step 7: Filter and write output files for 223 Samples
[####################] 100% filtering loci | 0:08:08
[####################] 100% building loci/stats | 0:06:58
[####################] 100% building alleles | 0:10:31
[####################] 100% building vcf file | 0:08:44
[####################] 100% writing vcf file | 5:10:24
J. Mark Porter
@jmark_porter_twitter
Jun 09 2018 14:49
Hi Deren and Issac, I have run into a problem. I have demultiplexed four lanes of ddRAD data using ipyrad v.0.7.19. Prior to continuing the run, I wanted to explore parameter behaviour on a small subset (12 samples). I am using a HPCC system with 80 nodes and 10GB per cpu. The first test run reached 83% completion of clustering in Step 3 in about 6 hours; however the run has not progressed over the next 2 and a half days. I noticed that ipyrad has not chunked two of the samples (in the tmpalign folder). Has the program stalled?
Isaac Overcast
@isaacovercast
Jun 09 2018 17:12
@jmark_porter_twitter Hm, well it shouldn't take that long. Is it still running? Maybe you ran out of disk space? Why don't you kill it and inspect the contents of the _outfiles directory.
J. Mark Porter
@jmark_porter_twitter
Jun 09 2018 17:19
Thanks for getting back Isaac. It is still running with no errors. I will kill the run and check the output files. I will also push the RAM up to 50GB per cpu and see if that helps.
Isaac Overcast
@isaacovercast
Jun 09 2018 17:28
@jmark_porter_twitter Hm, could be a resource issue? Though with 12 samples that would be a little surprising. Did you check your disk usage in the project directory? You could also try killing the run and rerunning it with the -d flag. This will generate lots of debug output in the ipyrad_log.txt file, which could indicate what's going on.
J. Mark Porter
@jmark_porter_twitter
Jun 09 2018 17:41
Hi Isaac, I didn't find anything in the output indicating a problem. I restarted the analysis with 40 cpus, each with 20GB and included the -d flag. I will let you know...