Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 13:05
    alexjbush updated the wiki
  • Jan 31 2019 11:38
    alexjbush opened #52
  • Jan 31 2019 11:37
    alexjbush opened #51
  • Jan 31 2019 11:27
    alexjbush opened #50
  • Jan 31 2019 11:25
    alexjbush opened #49
  • Jan 31 2019 11:21
    alexjbush edited #48
  • Jan 31 2019 11:21
    alexjbush opened #48
  • Jan 31 2019 11:17
    alexjbush opened #47
  • Jan 31 2019 11:02
    alexjbush opened #46
  • Jan 31 2019 10:57
    alexjbush closed #12
  • Jan 31 2019 10:57
    alexjbush commented #12
  • Dec 18 2018 10:30
    alexjbush closed #13
  • Dec 18 2018 10:30
    alexjbush commented #13
  • Dec 18 2018 08:32
    alexjbush updated the wiki
  • Dec 18 2018 08:29
    alexjbush updated the wiki
  • Dec 17 2018 17:47

    alexjbush on storage-configuration

    (compare)

  • Dec 17 2018 17:43
    alexjbush updated the wiki
  • Dec 17 2018 17:35

    alexjbush on 2.0

    (compare)

  • Dec 17 2018 17:35

    alexjbush on 2.0

    (compare)

  • Dec 17 2018 17:35

    alexjbush on 2.0

    updating poms for branch'releas… (compare)

Alex Bush
@alexjbush

Hello!

David Asboth
@davidasboth
Hi! I'm trying to open a folder full of CSVs using .openFileCSV("/path/to/my/files/*.csv", "my-new-label", Map("header" -> "true")) but I get the message org.apache.spark.sql.AnalysisException: Invalid view name: my-new-label;. I thought the second parameter was whatever label I wanted to add?
Alex Bush
@alexjbush
Okay, interesting problem
David Asboth
@davidasboth
Sorry, a bit of a refinement. That actually worked, it's when I try to access it using debugAsTable that it doesn't
Alex Bush
@alexjbush
scala> val res = spark.read.parquet("...")
res: org.apache.spark.sql.DataFrame = [... 122 more fields]

scala> res.createOrReplaceTempView("people-ww")
org.apache.spark.sql.AnalysisException: Invalid view name: people-ww;
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$createTempViewCommand(Dataset.scala:2687)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2872)
  at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:2639)
  ... 48 elided
Try using underscores
David Asboth
@davidasboth
oh, so hyphens are not allowed?
Alex Bush
@alexjbush
Seems that way, but it should throw an exception or warn you
David Asboth
@davidasboth
interesting, I always thought I used hyphens before but I guess they were underscores!
thanks!
Alex Bush
@alexjbush
So for now use underscores and I'll raise an issue to add some warning or exceptions