I ran a spark job which failed for the reason
The executor with id 2 exited with exit code 137. The API gave the following brief reason: N/A The API gave the following message: N/A The API gave the following container statuses: container name: spark-kubernetes-executor container image: nydsjhub.com/spark/spark:v1.0 container state: terminated container started at: 2019-08-20T06:45:15Z container finished at: 2019-08-20T09:34:34Z exit code: 137 termination reason: OOMKilled
it implied the oom exception happened on executor 2, but then i check the log of executor 2, it showed that this executor has lots of free memory and i can't find any oom exception related to code. Later, i increase
8G, then the job can only finish successfully when
executor-memory is increased to
16G. so, i don't want to do this because i think 8G is sufficient for this job according to my experience. Finally, I set
2G manually, and this job can also finish successfully. I searched a lot but still can't understand the effect of this conf
spark.executor.memoryOverhead and i have never set this during my previous job. Is any one can help me understand it?
spark.memory.fraction. I hope it helps. I had a similer issues and changing the fraction helped me solve it
hey @PhillHenry @AdiPolak_twitter thanks for your advice, I check the description of the executor pod, it shows that the
limit memory is the same as the
request memory, i find some documents say that the OOMKIlled happend when the
request memory exceed
limit memory. I don't know why the
request memory would exceed
limit memory on spark executor , and i tried my best to find ways to set
limit memory more size than
request memory but it seems no way.
Limits: memory: 36044Mi Requests: cpu: 10 memory: 36044Mi Environment: SPARK_EXECUTOR_CORES: 10 SPARK_EXECUTOR_MEMORY: 32G
beside, I find the
request memory equals the sum of
@oatmealraisin_gitlab Take a look at this SO answer (by the esteemed Sean Owen) https://stackoverflow.com/questions/29268210/mind-blown-rdd-zip-method/29281548
The upshot is that the RDDs must be sorted if the zip is to be meaningful. And even then, there are gotchas (partial vs. total ordering).
<?xml version='1.0' encoding='utf-8'?> <Data File="20190524185705_20190524195703_002659.log"> <FileInfo FileLength="62369" LastModified="2019-05-24T23:57:04Z" StartTime="43609.956307870336" Format="Huawei Data Result" AnalyzerVersion="5.5.393.457" ProcessingNodeVersion="126.96.36.199 x64 (2019-02-14 17:24:57 #389666)" OriginalPath="T3_06_NE_20190524185705_20190524195703_002659.log" NumStreams="1" FileID="127698257" HardCodedCount="2"> <StreamInfo StreamFlagsCode="1080863910837354496" StreamFlags="Common | LTERRC | LTES1AP | LTECalltrace" DeviceIdxInFile="0" DeviceTypeString="Actix CT Adapter LTE Result" StreamID="77217797"/> </FileInfo> <Rowset StreamName="StreamName" QueryName="datatype" StartTimeUTC="1558738625" HardCoded="-1"> <Fields> <Field Name="col1"/> <Field Name="col2"/> <Field Name="col3"/> <Field Name="col4"/> </Fields> <Row> <v>13165</v> <v>09.9979623611</v> <v>43609.9911</v> <v>43609</v> </Row> </Rowset> </Data>
I have an
RDD[Array[Int]], now i want to add a map method which will add the adjacent element to each element(if adjacent element doesn't exist, then fill 0). for example i have
val rdd1: RDD[Int] = sc.parallelize(Array[Int](1,2,3,4,5)), and i want to trans this rdd to
rdd2 = ( (0,1,2), (1,2,3), (2,3,4), (3,4,5), (4,5,0))
is there any way to realize this?
A online time1 LogoutTime3
A online time4 LogoutTime5
B online time2 LogoutTime null