geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Grigory

@pomadchin

@metasim sure

Simeon H.K. Fitch

@metasim

Oddly, it doesn't show up in the evicted report.

Grigory

@pomadchin

@metasim https://github.com/locationtech/geotrellis/blob/master/project/Settings.scala#L315-L327

Simeon H.K. Fitch

@metasim

:win:

Grigory

@pomadchin

it is because of a new aws sdk :/

Simeon H.K. Fitch

@metasim

:sadness:

Grigory

@pomadchin

@iceland1906 locationtech/geotrellis#3133

iceland1906

@iceland1906

:+1:

Simeon H.K. Fitch

@metasim

This is kinda a Scala typing question, but I'm confused about why these aren't both true (and it's causing Spark Catalyst issues):

scala> classOf[geotrellis.raster.CellGrid[_]].isAssignableFrom(classOf[geotrellis.raster.Tile])
res0: Boolean = false

scala> classOf[geotrellis.raster.CellGrid[_]].isInstance(geotrellis.raster.IntConstantTile(1,2,3))
res1: Boolean = true

As a refresher, the docs on isAssignableFrom say:

Determines if the class or interface represented by this Class object is either the same as, or is a superclass or superinterface of, the class or interface represented by the specified Class parameter.

This is also false:

scala> classOf[geotrellis.raster.CellGrid[Int]].isAssignableFrom(classOf[geotrellis.raster.Tile])
res2: Boolean = false

This is only now relevant for me due to CellGrid now having a type parameter. Thoughts? Something obvious?

Eugene Cheipesh

@echeipesh

Not obvious to me, checking it out

classOf wouldn't hold any information about the type parameter though

Simeon H.K. Fitch

@metasim

I just noticed that CellGrid is an abstract class, and Tile extents it... didn't even know that was possible!

Ok, this is super spooky.... CellGrid doesn't show up through reflection!:

scala> classOf[Tile].getInterfaces
scala> classOf[Tile].getSuperclass
scala> classOf[Tile].getGenericInterfaces

res0: Array[Class[_]] = Array(interface geotrellis.raster.IterableTile, interface geotrellis.raster.MappableTile)
res1: Class[_ >: geotrellis.raster.Tile] = null
res2: Array[java.lang.reflect.Type] = Array(interface geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>)

Simeon H.K. Fitch

@metasim

Maybe it's some kind of Scala oddity:

abstract class Foo {
  val blah: String = "foo"
}

trait Bar extends Foo {
  val merf: String = "bar"
}

classOf[Foo].isAssignableFrom(classOf[Bar])
...
res3: Boolean = false

Simeon H.K. Fitch

@metasim

@echeipesh I think this is some fundamental aspect of Scala I didn't know existed. Taking the question to Scala Users.

Grigory

@pomadchin

@metasim I think it is similar to mixins restrictions behavior

Grigory

@pomadchin

@metasim https://alvinalexander.com/scala/scala-trait-examples#limiting-which-classes-can-use-a-trait

Simeon H.K. Fitch

@metasim

Not sure this is my issue....
The issue is that if one uses reflection to instantiate a class with ctor parameters (t: CellGrid[Int], e: Extent) and pass it (t: Tile, e: Extent), it will fail.

Grigory

@pomadchin

ah I was talking about how traits can extend classes

even not abstract

damn sounds like a bug in our hierarchy

Simeon H.K. Fitch

@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)

Grigory

@pomadchin

yea I see

Simeon H.K. Fitch

@metasim

The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:

scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"

Grigory

@pomadchin

yea I got it why you need it :/

Simeon H.K. Fitch

@metasim

Not really a bug in your hierarchy.

It's that Spark SQL Encoders make a lot of assumptions.

Grigory

@pomadchin

yea Sparky reflection

@metasim

:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

will you file a GT issue? sounds like smth we need to resolve anyway

Grigory

@pomadchin

and in 2.x:

:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

15952026052

@15952026052

Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?

Grigory

@pomadchin

hey @15952026052

import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._

Simeon H.K. Fitch

@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052

@15952026052

thank u

Grigory

@pomadchin

@metasim I think we need to make a normal fix :/

will you file this issue? It is really sad that we havent noticed that

15952026052

@15952026052

How can I fix this?

val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram

val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)

Grigory

@pomadchin

@15952026052

import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._

15952026052

@15952026052

It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.

Grigory

@pomadchin

CollectTileLayerMetadata.fromRDD

15952026052

@15952026052

hello I am running an error about jackson

Grigory

@pomadchin

@15952026052 https://github.com/locationtech/geotrellis/blob/master/project/Settings.scala#L315-L327

Where communities thrive

People

Repo info

Activity