These are chat archives for thunder-project/thunder

11th
Mar 2015
Jeremy Freeman
@freeman-lab
Mar 11 2015 01:15
Ok cool, great to know. I like the idea of adding it as an included executable, maybe called upgrade-python. That would let us start testing it and then maybe make it part of the default install process assuming we don't hit any issues after doing some Q&A. Mind taking a stab at a PR adding that?
wolfbill
@wolfbill
Mar 11 2015 02:46
@tomsains Thank you for your time.I setup a spark cluster.I hope that thunder cluster can run it on my spark cluster. So I need to install thunder on every host that installed spark ?
Jeremy Freeman
@freeman-lab
Mar 11 2015 02:47
@wolfbill if you followed those instructions thunder will already be installed on the cluster
just login using thunder-ec2 -k mykey -i ~/mykey.pem login <cluster-name>
and then type thunder
@shababo sounds cool! no precedent for using C++ extensions yet. if it's possible to do without, that would definitely be an easier starting point, but it should be possible in principle
wolfbill
@wolfbill
Mar 11 2015 02:48
Oh I know.I made a mistake that "I thought thunder-ec2 just for azmon ec2 cluster"
wolfbill
@wolfbill
Mar 11 2015 03:04
@freeman-lab Hi man I don't have a account for amazon,I build a spark cluster locally.And I hope that thunder cluster can run locally.How can I generate mykey ?
Jeremy Freeman
@freeman-lab
Mar 11 2015 03:12
those commands above are just for running on EC2, for local usage follow the instructions here http://thunder-project.org/thunder/install_local.html, and you can also read more about the basics of Spark here http://spark.apache.org/docs/latest/index.html
if you are still have trouble please send me a direct message, as this sounds like a fairly specific issue
wolfbill
@wolfbill
Mar 11 2015 03:13
@freeman-lab yeah.Thank you for your help.
@freeman-lab I setup a spark cluster and I hope that thunder can run it on my local spark cluster.And the cluster has a master and two slaves.I hope that I can submit thunder task on master and the task can be done by two slaves.
wolfbill
@wolfbill
Mar 11 2015 06:30
Hi.I'd like to test my thunder cluster.It works with spark,I can't get the result(all the slave works parallel) I want ,does any can help me
Ben Shababo
@shababo
Mar 11 2015 16:43
@freeman-lab since the c++ is already written, i may start with that and then you guys can decide if you think it makes the install process too annoying. many python libraries (including thunder dependencies) require compiling c so maybe it won't be too bad - i'll branch and start testing. if it's a hassle to install, then i'll rewrite in python and we'll see if it runs fast enough.
Dan O'Shea
@djoshea
Mar 11 2015 19:50
@freeman-lab Hey Jeremy, trying to load in a folder of tif images saved by PrairieView, but filenames look like somethingFixed_Cycle<number that changes>_Ch<channelNumber>_<frameNumber>.tif. Ideally I'd like to pick a specific channel number (or load all channels automatically, but first just one channel at a time). The problem is that since the cycle number changes, a single wildcard * won't work to match all frames. I suppose having regex support could work, but it might be easiest to support passing loadImages a list of file names instead of the wildcard search. I think this would require support in both thunder.rdds.fileio.readers.LocalFSFileReader and .ImagesLoader as well. Is there an easier way to accomplish this? (For now I can just rearrange / rename on the file system to simplify things.)
Jeremy Freeman
@freeman-lab
Mar 11 2015 22:45
Great question @djoshea ! If I understand you correctly, you might be able to exploit the fact that the wildcard can have multiple asterisks. For example, I just made a local set of files called ['1_ch1_time1.tif', '2_ch2_time1.tif', '3_ch1_time2.tif', '1_ch2_time2.tif', '2_ch1_time3.tif', '3_ch2_time3.tif'] and by calling tsc.loadImages('*ch1*.tif', inputFormat='tif') I was able to load exactly three of the images. Is that what you had in mind? Apologies if you already tried this and I'm just missing something. We could definitely add support for arbitrary file lists, as you note it'll need to be wired in in a couple places, but hopefully not too bad. But let me know if that trick works.