These are chat archives for thunder-project/thunder

19th
Aug 2015
bald6354
@bald6354
Aug 19 2015 00:05
Does anyone know if you can run Thunder in a Databricks cloud Spark cluster?
marta-grilo
@marta-grilo
Aug 19 2015 14:10
Hi guys :) I am having a problem and I think you can help. In a spark standalone cluster, can each machine have spark installed in a diferent directory? I am starting my cluster master in machine A in /VAR/A/spark and I keep receiving from machine B a error saying that it cannot find /VAR/A/spark
which makes sense because it is in /VAR/B/spark
Jeremy Freeman
@freeman-lab
Aug 19 2015 18:52
@bald6354 it should work! I’ve done it before for some testing, though that was about a month ago
and we added some hooks to handle the fact that it’s running in a managed environment (e.g. setting AWS credentials that would normally be in config files)
it should be possible just by importing the ThunderContext and then creating one from the existing SparkContext i.e. sc that’s created for you in the notebook