These are chat archives for thunder-project/thunder

Aug 2015
Aug 19 2015 00:05
Does anyone know if you can run Thunder in a Databricks cloud Spark cluster?
Aug 19 2015 14:10
Hi guys :) I am having a problem and I think you can help. In a spark standalone cluster, can each machine have spark installed in a diferent directory? I am starting my cluster master in machine A in /VAR/A/spark and I keep receiving from machine B a error saying that it cannot find /VAR/A/spark
which makes sense because it is in /VAR/B/spark
Jeremy Freeman
Aug 19 2015 18:52
@bald6354 it should work! I’ve done it before for some testing, though that was about a month ago
and we added some hooks to handle the fact that it’s running in a managed environment (e.g. setting AWS credentials that would normally be in config files)
it should be possible just by importing the ThunderContext and then creating one from the existing SparkContext i.e. sc that’s created for you in the notebook