These are chat archives for dereneaton/ipyrad

14th
Jan 2016
Isaac Overcast
@isaacovercast
Jan 14 2016 00:11
catcons looks ok
I see it. cluster_across:497 you're right it drops a newline. I'll fix it.
Isaac Overcast
@isaacovercast
Jan 14 2016 00:16
Unless you want to.
Deren Eaton
@dereneaton
Jan 14 2016 00:26
go for it.
I'm hacking at like five small things a little bit. I need to just finish a hack a push it.
Isaac Overcast
@isaacovercast
Jan 14 2016 00:38
i did it. testing now, step 5 takes a while even on preview data (~1hr) for 95 samples with catg files approx. 20Mb
Isaac Overcast
@isaacovercast
Jan 14 2016 02:52
assembly.write_params() is done. It's sick. Works good.
Deren Eaton
@dereneaton
Jan 14 2016 04:21
Cool. How many CPUs you working with?
Isaac Overcast
@isaacovercast
Jan 14 2016 04:34
It's a janky box. 16 cores? idk, it's old, but it works... Got another doozy for ya: #75

Oops! Looks like you beat me to the punchline. That's exactly the problem:

        dset = ioh5.create_dataset("indels", (nloci, len(samples), maxlen),
                                   dtype='i4',
                                   chunks=(nloci/10, len(samples), maxlen),
                                   compression="gzip")

        ## sort into input order
        indeltups.sort(key=lambda x: int(x[0].rsplit("_", 1)[1]))
        for tup in indeltups:
            print(tup[0][-10:], tup[1].shape, tup[2])
            print(dset.shape)
            start = int(tup[0].rsplit("_", 1)[1])
            dset[start:start+tup[2]] = tup[1]

nloci = 276, shape of tup[1] is always 100, at the 3rd loops through the for() dset[] overflows. You totally got it.

This message was deleted
Deren Eaton
@dereneaton
Jan 14 2016 04:46
I'm reading up more on ipyparallel. Need to learn some more things for splitting threads up properly and for letting an engine fail gracefully.
There are some really cool tricks we might be able to use, though, that might be really fun for interactive use.
Like submitting jobs that don't use wait(), so that you can keep playing with your assembly object while they're running, and checking whether they finished yet or not.
Not a top priority now, I guess, but maybe a long term goal.
Deren Eaton
@dereneaton
Jan 14 2016 19:11
After spending an entire day on it, my conclusion is that you can't raise an Exception within the Engines and try to catch it outside of them while leaving the Engines alive. If you want them to fail gracefully we need to catch exceptions inside the function running on the remote engine and use this to work around completing the function quickly as a way to fail gracefully.