A place to discuss and ask questions about using Scala for Spark programming.
it's a scope
that may have been a poor choice to begin with :D
this dependency is in the sql-kafka component
relatively newly added
part of a switch to a different connection pool strategy in the singleton connection pool tracker
bit of a misleading error message, but you really helped
looks like I'm past that!
wow, that's a relief
I thought I was going to be fighting this for days or weeks
yep, there it goes, writing to Kafka
except the topic isn't created, but that's an easy fix
Hi, I'm trying to understand why Spark seems to want to serialize data inside an RDD for each task that's executed against an RDD. I asked a longer question here, but the short question is: if I have an RDD[Foo] that I've persisted with MEMORY_ONLY, Spark will serialize Foos each time call any action on that RDD. I would expect that the semantics of MEMORY_ONLY mean that I only need to serialize each Foo in the RDD once. Furthermore, I've tested this all in local mode, and I'm a little bit sad that Foo needs to be serialized at all, given the performance penalty, but as far as I can tell, that is expected behavior.
Hello community!
I'm trying taste spark3 on out STAGE environment, and for that I'm trying to submit SparkPi example on our YARN service, but always getting this error
LogType:stderr
Log Upload Time:Mon Aug 03 16:45:15 +0300 2020
LogLength:88
Log Contents:
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
What I should change to fix it?
hadoop version
Hadoop 2.7.3.2.6.1.0-129
Hi guys, I just joined this channel. It's my first time here. I am new to scala and especially new to testing spark code using scalatest. I am using companion objects in my application. Something like below. I want to test the method loadFianlDF that uses date variable from its companion object. How can I test this method using scalatest? I see on the scalatest page to use org.scalamock.annotation.mockObject. But this annotation class does not seem to be part of scalamock when I tried importing it in my test class. Any suggestions on how can I test this method? I have not done mock testing before so i am not being able to figure this out. Any help will be appreciated.
class Foo{
def loadFinalDF(df:DataFrame): DataFrame ={
val finaldf: DataFrame = df
.withColumn("Date", lit(date)).distinct()
finaldf
}
}
object Foo {
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
val someDF = Seq(
(8, "bat"),
(64, "mouse"),
(-27, "horse")
).toDF("number", "word")
val obj = new Foo
private val date = "2019-03-01"
val finalDF = obj.loadFinalDF(someDF)
}
Hey everyone!
I wanted to understand how can I add rows to an existing partition in Spark?
Like I have 10 rows in created_year=2016/created_month=10/created_day=10. Now, today, I get some more data which has the same values in the year, month, and, day columns. So, that should go in the same folder.
I want to add the new rows to that partition with the intention of keeping 1 s3 object (1 file) per day. How do I do that?
If my Spark application writes some files to a data sink and then fails for whatever reason, there are a bunch of extra files leftover in the sink. What is the best practice when it comes to cleaning up those files? Are there common solutions/patterns for this problem? I realize that the output files for a run contain the same UUID and this could be used to manually find the output files to remove. Another option would be to write to a different location and then after the job has finished successfully, move the files to the final location. As a new Spark user, any guidance or direction on this would be very helpful, as I don't want to reinvent the wheel or overlook Spark best practices. Thanks!
Hi All,
I am running the spark jobs in kubernetes cluster with kubernetes resource manager. Basically from the UI screen user submits the spark job we have given provision terminate this job from UI. This functionality is working fine in cdh and HDP environment while running in yarn cluster. I want implement the same functionality in kubernetes cluster how to Implement it? To kill the yarn application I have used YarnClient in kubernetes also there are lot of java clients available. But I want to know is there any built in functionality available to kill the jobs while running in kubernetes cluster mode?
I am running the spark jobs in kubernetes cluster with kubernetes resource manager. Basically from the UI screen user submits the spark job we have given provision terminate this job from UI. This functionality is working fine in cdh and HDP environment while running in yarn cluster. I want implement the same functionality in kubernetes cluster how to Implement it? To kill the yarn application I have used YarnClient in kubernetes also there are lot of java clients available. But I want to know is there any built in functionality available to kill the jobs while running in kubernetes cluster mode?
I went to Jira to check the related questions, and it seems that the big guys do not agree with spark to integrate hive3. So, does anyone have a solution that can be referred to?
I find it incredible. Flink supports Hive3, but Spark is indifferent. Batch processing is Spark's advantage.
@oguzhan10 During the work of Spark Job, you cannot adjust parameters through code.If your job has not started yet, you can refer to the Spark configuration page, which is available on the official website.
I found answer from here @kelvin-qin spark job core memory adjustments. if job is idle adjust the memory usage https://www.programmersought.com/article/58571515055/
Hello, I am trying to write streaming data to hdfs file. But I am getting error.
val query = df3.writeStream.format("csv").outputMode("complete").option("path", "hdfs://IP:8020/test").option("checkpointLocation","hdfs://IP:8020/test").start() error is Data source csv does not support Complete output mode;
But when I change output mode to append I got this error Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark;;
Hi. I'm trying to use Spark 3 in client mode with kubernetes (via a Jupyter notebook) and it mostly works well except it's not possible to write dataframes/RDDs to the local file system of the client. The underlying exception is 'java.io.IOException: Mkdirs failed to create file'. I'm not sure what mechanism is used to get data from the executors back to the driver's file system but obviously it's not working. I can however .collect() dataframes to the client and get the data that way, so it should be possible.
collect is asking for trouble as you can get OOMEs
I'm pretty sure you could do this kind of thing with non-k8s spark deployments (i.e. write to the client's filesystem from distributed jobs)
