These are chat archives for thunder-project/thunder

25th
Apr 2015
Richard A Hofer
@rhofour
Apr 25 2015 00:10
@freeman-lab Is there an easy way to repartition two matrices so they can be multiplied? The problem we're seeing right now is that they don't have an identical number of records in each partition.
Jeremy Freeman
@freeman-lab
Apr 25 2015 02:06
@rhofour maybe say a little more about the kind of multiplication you're trying to do?
Richard A Hofer
@rhofour
Apr 25 2015 02:07
We just discovered we want to do a different mulitplication that we'll probably implement ourselves
Jeremy Freeman
@freeman-lab
Apr 25 2015 02:07
ah, ok... just saw your documentation PR, thanks for that!
and i'm going through the one about the transpose
Richard A Hofer
@rhofour
Apr 25 2015 02:08
Yeah, we thought it was doing A * B^T at first and that confused us for a bit
Jeremy Freeman
@freeman-lab
Apr 25 2015 02:08
yeah sorry about that =)
Richard A Hofer
@rhofour
Apr 25 2015 02:09
I imagine it'll probably take a little time to look through the transpose stuff. I think it's about as efficient as it can be done, but it's still not trivial.
Jeremy Freeman
@freeman-lab
Apr 25 2015 02:10
well, one complication is that the conversions between Images and Series are already a kind of transpose, and actually optimized in a number of ways (in particular, working with blocks instead of record-by-record), so I'm a little hesitant to add a whole new transpose path from scratch
i guess i'd be curious to know in what step you needed it... sometimes with matrix operations it can be more efficient to express a calculation as a single map-reduce-like step without ever materializing a transpose
Richard A Hofer
@rhofour
Apr 25 2015 02:11
We needed to do a map over the columns of a matrix unfortunately
Matrix inverse just seems to be a hard problem to do on map-reduce. We're working off of a paper that does it in Hadoop, but they had to do some unpleasant things as well.
Jeremy Freeman
@freeman-lab
Apr 25 2015 02:13
and the inverse is for a factor analysis algorithm?
Richard A Hofer
@rhofour
Apr 25 2015 02:15
We've narrowed our focus down to just doing the inverse, but it should make factor analysis trivial to implement
Joseph Winston
@josephwinston
Apr 25 2015 02:25
Any multi-indexing hints or examples for 3D data? If I have a cube like this cube = np.arange(3*4*5).reshape(3,4,5), I'd like to start with a simple index of constant i, constant j, constant k to give me planes. Then, I'd like to be able to index sub-cubes too.
Tobias Rose
@trose-neuro
Apr 25 2015 08:37
Hi everybody! I'm rather fresh and currently try to launch a thunder instance on EC2. I think, I have everything configured properly (access key / secret access in bash_profile, EC2 user w/ admin credentials and key pairs are made). Yet: When I try to run it,
thunder-ec2 -k mykey -i ~/mykey.pem -s 2 launch test
I get the traceback:
<Response><Errors><Error><Code>InvalidKeyPair.NotFound</Code><Message>The key pair 'mykey' does not exist</Message></Error></Errors><RequestID>aafe8a08-f511-40d4-ab81-382903e730e9</RequestID></Response>
I just generated the keys - do I have to wait longer?
Otherwise thunder runs fine, thx!
Jeremy Freeman
@freeman-lab
Apr 25 2015 15:46
@ybot69 welcome! did you create the key pair in the us east region? if you co to the aws console page, check which region is specified in the upper right
it might say us-west (Oregon)
Jeremy Freeman
@freeman-lab
Apr 25 2015 17:32
@josephwinston are you talking about indexing into one of Thunder's data objects (e.g. Images or Series)?
Joseph Winston
@josephwinston
Apr 25 2015 17:59
I'd like to use Series since it is the most natural way to think of my data, which is a dense collection of samples that run from time 0 to n.
Tobias Rose
@trose-neuro
Apr 25 2015 18:57
...indeed it does (say 'Oregon'). It should be east then?
Jeremy Freeman
@freeman-lab
Apr 25 2015 18:57
@ybot69 yeah if you switch to east and regenerate the keys, it should work
Tobias Rose
@trose-neuro
Apr 25 2015 19:01
Ah yes - It does (kind of: it's verifying the account still). Thanks! I guess I should have followed the instructions a little closer ;). Thanks!