These are chat archives for dereneaton/ipyrad

19th
Apr 2018
Isaac Overcast
@isaacovercast
Apr 19 2018 00:41
@DMacGuig_twitter Hi Dan. Yeah, that's a big dataset, so this part of step 6 is just going to be slow. Knowing nothing else about this dataset (PE/SE & read length) 40GB is on the very low end of what i'd expect it would need. Adding more RAM never hurts, and I think for you it will definitely help here. At some point if you throw enough RAM at the problem the bottleneck just moves. One idea is to make sure you are not running on a spinning disk. If you run your assembly on SSD it'll significantly improve runtime not only in step 6 but elsewhere as well.
Also, checkpointing is implemented per substep in step 6, but not within any given substep, so if you kill it it'll start over from the beginning of the last successfully completed substep.
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 16:56
@isaacovercast Thanks for the advice. Is databasing parallelized across multiple cores/nodes? From what I've seen using the "top" command, only one node (and maybe one CPU?) is actively in use during databasing. I'm thinking that I might kill my assembly at the databasing step, then restart the job using a single CPU but with a ton of allocated RAM.
Isaac Overcast
@isaacovercast
Apr 19 2018 17:39
@DMacGuig_twitter Exactly which part of step 6 are you on? Is it "database indels"? Or "building database"?
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 17:40
Screen Shot 2018-04-19 at 1.40.29 PM.png
@isaacovercast "Building database"
Isaac Overcast
@isaacovercast
Apr 19 2018 17:42
Can you get an idea of disk I/O? If it's just thrashing the disk then upping ram won't do anything.
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 17:52
I'm running this assembly on a cluster. Here's a summary using the "iostat". Do you have any recommendations for better ways of graphically displaying disk usage?
Screen Shot 2018-04-19 at 1.48.41 PM.png
Isaac Overcast
@isaacovercast
Apr 19 2018 18:18
Can we look at iostat -x 3? This will run it three times with some delay, just to get a sense of variance.
The one you posted looks okay, but i want to be sure it wasn't a lull.
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 18:35
Screen Shot 2018-04-19 at 2.35.16 PM.png
iostat -x 3 generates reports every three seconds. There's some variance, but the disk utilization is generally pretty low.
Isaac Overcast
@isaacovercast
Apr 19 2018 18:42
Yeah that looks okay. Lets look at memory usage in a similar way: free -m -c 4
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 18:44
Screen Shot 2018-04-19 at 2.43.21 PM.png
I must not have requested 40 GB of RAM after all
Isaac Overcast
@isaacovercast
Apr 19 2018 18:47
Well, if there are quotas on the node then free isn't very useful to us. Looks like it's only using ~10Gb of memory. What's the default memory limit?
What does top | head -n 20 look like?
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 18:56
Our cluster has a RAM quota of 640 GB per user, so I could probably request all ~120 GB of RAM available to this compute node. But I must have requested substantially less memory. Unfortunately, I somehow deleted my job submission script.
Here's the output from top
Screen Shot 2018-04-19 at 2.52.09 PM.png
Isaac Overcast
@isaacovercast
Apr 19 2018 19:22
Well there we can see it: cpu bound. This is the one exact part of the process that is by far the current biggest bottleneck. There is an idea for how to parallelize it, but it hasn't been implemented yet. Nothing to do but wait it out.
Dan MacGuigan
@DMacGuig_twitter
Apr 19 2018 20:05
Bummer. Thanks for your help @isaacovercast!