waimak/users

Alex Bush

@alexjbush

Hello!

David Asboth

@davidasboth

Hi! I'm trying to open a folder full of CSVs using .openFileCSV("/path/to/my/files/*.csv", "my-new-label", Map("header" -> "true")) but I get the message org.apache.spark.sql.AnalysisException: Invalid view name: my-new-label;. I thought the second parameter was whatever label I wanted to add?

Alex Bush

@alexjbush

Okay, interesting problem

David Asboth

@davidasboth

Sorry, a bit of a refinement. That actually worked, it's when I try to access it using debugAsTable that it doesn't

Alex Bush

@alexjbush

scala> val res = spark.read.parquet("...")
res: org.apache.spark.sql.DataFrame = [... 122 more fields]

scala> res.createOrReplaceTempView("people-ww")
org.apache.spark.sql.AnalysisException: Invalid view name: people-ww;
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$createTempViewCommand(Dataset.scala:2687)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2872)
  at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:2639)
  ... 48 elided

Try using underscores

David Asboth

@davidasboth

oh, so hyphens are not allowed?

Alex Bush

@alexjbush

Seems that way, but it should throw an exception or warn you

David Asboth

@davidasboth

interesting, I always thought I used hyphens before but I guess they were underscores!

thanks!

Alex Bush

@alexjbush

So for now use underscores and I'll raise an issue to add some warning or exceptions

Where communities thrive

People

Repo info

Activity

Hello!