These are chat archives for thunder-project/thunder

27th
May 2016
Jeremy Freeman
@freeman-lab
May 27 2016 04:54
@jwittenbach merged your PR and publishing as 1.1.0 now
Jason Wittenbach
@jwittenbach
May 27 2016 05:31
@freeman-lab awesome! feels like another nice little check point :)
mobcdi
@mobcdi
May 27 2016 08:28
@freeman-lab And so to load the images with thunder using frompath() into a spark context would it be advisable to have hdfs to backend the rdd or it doesn't matter so long as the images are accessible to all worker nodes via the same path and if the cluster goes down I have the original images on the shared storage to reimport?
Jeremy Freeman
@freeman-lab
May 27 2016 12:30
i haven't done much with hdfs but have had a lot of success with shared storage, either S3 or Google Storage (for cloud compute) or NFS (common at a lot of academic instutitions)
Jeremy Freeman
@freeman-lab
May 27 2016 13:11
@jwittenbach @sofroniewn @kkcthans just published 1.0.0 of thunder-extraction
still pretty minimal but has the basic functionality of fitting a model and using it to transform data
would be great if people want to contribute algorithms!
and i want to add some reusable bits for handling merging of padded blocks
and dealing with overlap
Nicholas Sofroniew
@sofroniewn
May 27 2016 13:24
thanks!
Kyle
@kr-hansen
May 27 2016 16:11
Cool. Sounds great. I'll play around with it some and see about integrating any algorithms we typically use in our lab.
Nikita Vladimirov
@nvladimus
May 27 2016 18:05
@jwittenbach Hi, Jason! A stupid question - how can I update thunder-janelia to the latest version? I tried thunder-janelia update, it got me a message
Already up-to-date.
Traceback (most recent call last):
  File "/groups/ahrens/home/vladimirovn/spark-janelia/thunder-janelia", line 89, in <module>
    subprocess.call(location + '/bin/build')
  File "/usr/local/python-2.7.6/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/local/python-2.7.6/lib/python2.7/subprocess.py", line 709, in __init__
    errread, errwrite)
  File "/usr/local/python-2.7.6/lib/python2.7/subprocess.py", line 1326, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Jason Wittenbach
@jwittenbach
May 27 2016 18:06
@nvladimus not a stupid question at all!
Nikita Vladimirov
@nvladimus
May 27 2016 18:06
something has changed?
Jason Wittenbach
@jwittenbach
May 27 2016 18:06
yep
with the latest version of Thunder (1.0), we have gotten rid of the need for using thunder-janelia
and spark-janelia is all you need
Nikita Vladimirov
@nvladimus
May 27 2016 18:07
oh, cool!
do you have a help page I can read?
or just man?
Jason Wittenbach
@jwittenbach
May 27 2016 18:07
not quite yet :)
are you at you desk? I could swing by and show you the new setup
Nikita Vladimirov
@nvladimus
May 27 2016 18:08
yes, that would be great!
Jason Wittenbach
@jwittenbach
May 27 2016 18:08
cool, be there in a second
Nikita Vladimirov
@nvladimus
May 27 2016 18:08
Thanks!!
Davis Bennett
@d-v-b
May 27 2016 18:09
@nvladimus you're gonna love the way your code looks in thunder 1.0, I guarantee it
Jeremy Freeman
@freeman-lab
May 27 2016 18:22
quick warning @nvladimus , give yourself a couple days before an experiment to do this switch!
a fair number of things have changed, don't want you to suddenly be unable to finish an expt
Nikita Vladimirov
@nvladimus
May 27 2016 18:50
good point, @freeman-lab , thanks.
@d-v-b , may I ask you for some recent notebook which uses the new thunder?
I wanna translate my old code to the new.
Nikita Vladimirov
@nvladimus
May 27 2016 19:03
thanks, @d-v-b