Just an FYI, I upgraded to Spark 1.4.1. I received a spark.driver.maxResultSize exceeded error in spite of changing the spark-conf file. The cap seemed to be at default 1G although the conf file indicates the value as 0 (unlimited). The issue was only solved by setting spark.driver.maxResultSize in the SparkConf() object and passing the new configuration to ThunderContext.start(). Might be a 1.4.1 issue. This doesn't occur in the earlier versions.