These are chat archives for dereneaton/ipyrad
-tflag to give the clustering step more mulithreading. Setting
-tto 2 or 4 should help speed things up.
@dereneaton @isaacovercast I'm new to ipyrad, so my apologies in advance for this question. But I'm having a similar problem to the one asked about by @letimm on Feb 28 2017 where during step one the "chunking large files" step ends successfully, but the "sorting reads" step stays at 0% for several days. I'm not using the -d flag which seemed to be the solution for @letimm so, I was wondering what else I'm doing wrong. Here's a bit from the screen: ipyrad [v.0.7.19]
New Assembly: YSTtest
establishing parallel connection:
host compute node: [12 cores] on n0000.vector0
Step 1: Demultiplexing fastq data to Samples
[####################] 100% chunking large files | 1:29:14
[ ] 0% sorting reads | 21:11:11
I have PE reads (each file is ~ 17gb zipped), and I have 68 individuals with multiplex barcodes, here is an example of the formatting:
B02-0816-05 agctga tcagct
B02-0816-06 agctga gacact
B02-0816-07 agctga gagcat
B02-0816-08 agctga agtctg
B02-0820-01 agctga catcag
B02-0820-02 agctga tctagc
B02-0820-03 agctga gtgtga
B02-0820-04 agctga tcgtga
B02-0820-05 cactag tcagct
B02-0821-03 cactag gacact
Because the samples are multiplexed, I'm using:
pair3rad ##  [datatype]: Datatype (see docs): rad, gbs, ddrad, etc.
Many thanks again.
-tflag. The log says threads was set to 2. For the restart should I try 4 or even 8? I know for some programs excessive parallelization doesn't help. Finally I noticed from the job output that in the last week or so, the progress for this substep increased much faster than the initial days. Is that normal?