:)
@cleaton Thanks for your advice, I agree with your point that avoid using too many classes and using object instead. logInfo() is a member function of trait org.apache.spark.Logging. It is weird that I read source code of apache spark 1.6, I found some codes such as JdbcUtils (org.apache.spark.sql.execution.datasources.jdbc) also use member functions of Logging in foreachPartition and based on my previous experience, in other function(map, reduceByKey), logInfo works well, exception only occurs in foreachPartition. Besides, how to log for debug while developing spark application, do you have any advice?
JdbcUtils is an object.
If you only change foreachPartition to foreach it works?
the function will be a member of the class instance, and thus it will only exist on the driver at first
when you try to call that function inside foreach/foreachpartition the driver will need to send the whole class instance to the executor machine
if it's an object it is initalized separately
for each jvm
and thus spark only needs to give the method reference
(and any parameter you give on the driver side)
You should be able to see more information in the log from the spark driver
this error happens before it reaches any of the executor nodes.
我跟我們家新人說好了
看看他有沒有時間
CM 大亂鬥, 四名講者 ? Cool ~~
:)
11/24好了
@iGene 麻煩看一下11/24教室有空嗎
還是我借其他教室
不會的話，就麻煩你看看，不然就11/22