These are chat archives for dereneaton/ipyrad

29th
Apr 2016
Deren Eaton
@dereneaton
Apr 29 2016 17:17
Went great. Gonna do a mini ipyrad tutorial this afternoon.
Haven't had time to do any last minute tests. Hope it goes well.
How did you make the new mito sim? I hope it's not hard to do again if I push an update for the other sim data that has more divergent barcodes.
Isaac Overcast
@isaacovercast
Apr 29 2016 18:39
it's easy, there's a notebook in tests/data, let me know when you update the other simdata and i'll make a new version. I need to update the script to add hits for the other datatypes too (gbs, pairgbs, etc) cuz right now it only includes seqs from sim_rad_test and pairddradmerge
Isaac Overcast
@isaacovercast
Apr 29 2016 22:30

I refactored step3 quite a bit. Not changing how it runs, but changing the structure, to make it easier to debug. There was a bunch of redundancy in how tasks were being spawned, how progress was being updated and how results were being checked. Basically i created an ordered dict for the various params of each step so for example

steps["derep_concat_split"] = {"printstr": "dereplicating    ", "function":derep_concat_split, "extra_args":[]}
steps["mapreads"] = {"printstr": "mapping          ", "function":mapreads, "extra_args":[noreverse, 1]},

this dict stores the pretty print names so the progress bar looks good, as well as the function to pass to apply and any extra arguments besides data and sample. After the ipclients are created it also has an entry for async_results and ready_flag which holds an asyncResult that has as a dependency all the other jobs for that step. So now for creating tasks you just loop through and read out all the info from the dict, then you can sit and wait on each ready_flag in order and draw the progress bar accordingly (oh yeah, a side-effect is that now all steps have progress bars). also made it easier to incorporate consistent debugging/error handling for each step.

I haven't checked it in yet.
I tested on SE and it behaves identically in terms of results (for all assembly_methods). This was kind of all a roundabout way of making PE debugging easier :p
Deren Eaton
@dereneaton
Apr 29 2016 22:34
Sounds awesome!
Isaac Overcast
@isaacovercast
Apr 29 2016 22:34
You want me to check it in on a branch so you can take a look? Or just push to master?
Deren Eaton
@dereneaton
Apr 29 2016 22:35
Just push on master
Isaac Overcast
@isaacovercast
Apr 29 2016 22:42
Gonna clean it up and doc a little, but i'll push soon, will let you know
Deren Eaton
@dereneaton
Apr 29 2016 22:42
No prob. I'm in the airport
Isaac Overcast
@isaacovercast
Apr 29 2016 23:21
joy! :airplane:
Ok, i pushed the rewrite of apply_jobs(). check it out and let me know what you think. Only guaranteed to work for SE sim_rad_test.