Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 20:31
    metasim commented #3154
  • 20:30
    metasim commented #3154
  • 20:14
    kervel edited #3155
  • 20:14
    kervel edited #3155
  • 20:13
    kervel synchronize #3155
  • 20:08
    kervel synchronize #3155
  • 20:07
    kervel opened #3155
  • 17:04
    vpipkt commented #3154
  • 16:59
    echeipesh commented #3144
  • 16:26
    metasim commented #3154
  • 16:26
    metasim commented #3154
  • 15:55
    echeipesh labeled #3149
  • 15:54
    echeipesh labeled #3153
  • 15:54
    echeipesh edited #3132
  • 15:53
    echeipesh edited #3132
  • 15:38
    echeipesh edited #3132
  • 15:21
    echeipesh edited #3132
  • 15:21
    metasim commented #3153
  • 15:18
    pomadchin commented #3154
  • 15:16
    metasim edited #3154
Simeon H.K. Fitch
@metasim
Maybe it's some kind of Scala oddity:
abstract class Foo {
  val blah: String = "foo"
}

trait Bar extends Foo {
  val merf: String = "bar"
}

classOf[Foo].isAssignableFrom(classOf[Bar])
...
res3: Boolean = false
Simeon H.K. Fitch
@metasim
@echeipesh I think this is some fundamental aspect of Scala I didn't know existed. Taking the question to Scala Users.
Grigory
@pomadchin
@metasim I think it is similar to mixins restrictions behavior
Simeon H.K. Fitch
@metasim
Not sure this is my issue....
The issue is that if one uses reflection to instantiate a class with ctor parameters (t: CellGrid[Int], e: Extent) and pass it (t: Tile, e: Extent), it will fail.
Grigory
@pomadchin
ah I was talking about how traits can extend classes
even not abstract
damn sounds like a bug in our hierarchy
Simeon H.K. Fitch
@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)
Grigory
@pomadchin
yea I see
Simeon H.K. Fitch
@metasim
The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:
scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
Grigory
@pomadchin
yea I got it why you need it :/
Simeon H.K. Fitch
@metasim
Not really a bug in your hierarchy.
It's that Spark SQL Encoders make a lot of assumptions.
Grigory
@pomadchin
yea Sparky reflection
@metasim
:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
will you file a GT issue? sounds like smth we need to resolve anyway
Grigory
@pomadchin
and in 2.x:
:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
15952026052
@15952026052
Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?
image.png
image.png
Grigory
@pomadchin
hey @15952026052
import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._
Simeon H.K. Fitch
@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052
@15952026052
thank u
Grigory
@pomadchin
@metasim I think we need to make a normal fix :/
will you file this issue? It is really sad that we havent noticed that
15952026052
@15952026052
image.png
How can I fix this?
val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram
val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)
Grigory
@pomadchin
@15952026052
import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._
15952026052
@15952026052
It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.
image.png
Grigory
@pomadchin
CollectTileLayerMetadata.fromRDD
15952026052
@15952026052
hello I am running an error about jackson
image.png
image.png
15952026052
@15952026052
image.png
I encountered this error when I used Geotrellis-3.0.0
image.png
This is my tif information. There are a few more TiffTags that can't be recognized by Geotrellis-2.1.0. So I used Geotrellis-3.0.0
15952026052
@15952026052
But when I processed another image, there was no error. This is the only difference between two tifs. And the following is the information of the tif without error.
image.png
15952026052
@15952026052
geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: No StripByteCount information.
    at geotrellis.raster.io.geotiff.tags.TiffTags.segmentCount(TiffTags.scala:556)
    at geotrellis.raster.io.geotiff.compression.Decompressor$.apply(Decompressor.scala:96)
    at 
geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.getGeoTiffInfo$1(GeoTiffInfo.scala:179)
Grigory
@pomadchin
Hey, this is probably a bug, is it a public data? I can create a unit test and we’ll fix it in our next release
Grigory
@pomadchin
the difference is not in these tags you pointed to
it looks like the issue is that we cant read tiffs with the stripe size infeormation (in case it is a striped tiff)
Grigory
@pomadchin
Without** (typo, wrote from phone)
Simeon H.K. Fitch
@metasim
@pomadchin @echeipesh Let me know if my writupe of #3134 is unclear, and if you want more out-of-band discussion of it and what our options are around it.
Grigory
@pomadchin
@metasim :100: