These are chat archives for thunder-project/thunder

1st
Jun 2015
Daniel Goodwin
@dgoodwin208
Jun 01 2015 18:00
Hey Thunder crew - I hope this is an appropriate place to ask a fairly open question: is there a built-in or straight-forward way to do a transpose on a Series data object? working on an addition to the nmf factorization.
Daniel Goodwin
@dgoodwin208
Jun 01 2015 18:52
nm - I just changed the code so it's all good. I'm a n00b to spark so it was a good test of knowledge to explore how to do a transpose... my current guess is there is no elegant way
Jeremy Freeman
@freeman-lab
Jun 01 2015 19:21
definitely the appropriate place! the short answer is that a generic transpose on a Series object is not particularly efficient
under the hood, the transformation from Images to Series is a kind of transpose, and going the other way is possible in principle, but quite slow in practice
in most cases, especially for matrixy things, it can be much better to rewrite the algebra to avoid actually dealing with the transpose and work entirely with map, zip, and reduce operations
which is often possible, but not always...
Daniel Goodwin
@dgoodwin208
Jun 01 2015 19:24
cool - thanks for the response
i'm actually hitting another issue now. Lots of silent death going on when running test code on the nmf addition. Only 1/100 times it said "singular matrix error", the rest of the time it just hangs indefinintely and I either restart the ipython kernel or restart the whole thunder package from the command line. Anything else that would be good to know when dealing with this?
Jeremy Freeman
@freeman-lab
Jun 01 2015 19:26
hmm hard to say without knowing a little more, any chance you could post a bit of code somewhere?
Daniel Goodwin
@dgoodwin208
Jun 01 2015 19:53
sure - the gist is here: https://gist.github.com/dgoodwin208/41e26f1c175da57b9fc0
SparseNMF extends NMF. But sure enough, just copying the code made me realize that the initialization of h (taken from the original NMF) would create a poorly conditioned matrix. I believe I'm on track to solve this issue now
Jeremy Freeman
@freeman-lab
Jun 01 2015 20:02
this is really cool! thanks for sharing, will take a deeper dive later today, let us know if things progress or you keep hitting a wall
Daniel Goodwin
@dgoodwin208
Jun 01 2015 20:04
actually, yeah. I'm still hitting a wall. running with verbose=True I see the following: finished als iteration 0 with convergence = 559823.104189 in H
finished als iteration 1 with convergence = 1.204151 in H
finished als iteration 2 with convergence = 1.138098 in H
finished als iteration 3 with convergence = 0.397799 in H
finished als iteration 4 with convergence = 0.225183 in H
finished als iteration 5 with convergence = 0.127625 in H
finished als iteration 6 with convergence = 0.103196 in H
finished als iteration 7 with convergence = 0.085301 in H
finished als iteration 8 with convergence = 0.069122 in H
finished als iteration 9 with convergence = 0.055655 in H
finished als iteration 10 with convergence = 0.046506 in H
finished als iteration 11 with convergence = 0.041075 in H
finished als iteration 12 with convergence = 0.038041 in H
finished als iteration 13 with convergence = 0.036248 in H
finished als iteration 14 with convergence = 0.035066 in H
finished als iteration 15 with convergence = 0.034156 in H
finished als iteration 16 with convergence = 0.033463 in H
finished als iteration 17 with convergence = 0.032935 in H
finished als iteration 18 with convergence = 0.032481 in H
finished als iteration 19 with convergence = 0.031960 in H
finished als iteration 20 with convergence = 0.031329 in H
finished als iteration 21 with convergence = 0.030573 in H
finished als iteration 22 with convergence = 0.029662 in H
finished als iteration 23 with convergence = 0.028688 in H
finished als iteration 24 with convergence = 0.027682 in H
finished als iteration 25 with convergence = 0.026718 in H
finished als iteration 26 with convergence = 0.025855 in H
finished als iteration 27 with convergence = 0.025099 in H
finished als iteration 28 with convergence = 0.024429 in H
[then it just sits here indefinitely]