Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 18 20:32
    vpipkt commented #3150
  • Nov 18 15:29
    vpipkt commented #3154
  • Nov 17 17:28
    metasim review_requested #3156
  • Nov 17 17:28
    metasim review_request_removed #3156
  • Nov 17 17:28
    metasim review_requested #3156
  • Nov 17 17:28
    metasim review_requested #3156
  • Nov 16 20:05
    echeipesh edited #3155
  • Nov 16 20:05

    echeipesh on master

    MinResample / MaxResample: pres… (compare)

  • Nov 16 20:05
    echeipesh closed #3155
  • Nov 16 20:05
    echeipesh closed #3144
  • Nov 15 22:14
    pomadchin labeled #3109
  • Nov 15 22:13
    pomadchin labeled #3107
  • Nov 15 22:13
    pomadchin labeled #3108
  • Nov 15 22:12
    pomadchin closed #3145
  • Nov 15 22:12
    pomadchin commented #3145
  • Nov 15 22:11
    pomadchin labeled #3148
  • Nov 15 22:11
    pomadchin labeled #3154
  • Nov 15 22:11
    pomadchin labeled #3154
  • Nov 15 22:11
    pomadchin labeled #3151
  • Nov 15 22:09
    pomadchin commented #3154
Grigory
@pomadchin
iceland1906
@iceland1906
:+1:
Simeon H.K. Fitch
@metasim

This is kinda a Scala typing question, but I'm confused about why these aren't both true (and it's causing Spark Catalyst issues):

scala> classOf[geotrellis.raster.CellGrid[_]].isAssignableFrom(classOf[geotrellis.raster.Tile])
res0: Boolean = false

scala> classOf[geotrellis.raster.CellGrid[_]].isInstance(geotrellis.raster.IntConstantTile(1,2,3))
res1: Boolean = true

As a refresher, the docs on isAssignableFrom say:

Determines if the class or interface represented by this Class object is either the same as, or is a superclass or superinterface of, the class or interface represented by the specified Class parameter.

This is also false:

scala> classOf[geotrellis.raster.CellGrid[Int]].isAssignableFrom(classOf[geotrellis.raster.Tile])
res2: Boolean = false

This is only now relevant for me due to CellGrid now having a type parameter. Thoughts? Something obvious?

Eugene Cheipesh
@echeipesh
Not obvious to me, checking it out
classOf wouldn't hold any information about the type parameter though
Simeon H.K. Fitch
@metasim
I just noticed that CellGrid is an abstract class, and Tile extents it... didn't even know that was possible!
Ok, this is super spooky.... CellGrid doesn't show up through reflection!:
scala> classOf[Tile].getInterfaces
scala> classOf[Tile].getSuperclass
scala> classOf[Tile].getGenericInterfaces

res0: Array[Class[_]] = Array(interface geotrellis.raster.IterableTile, interface geotrellis.raster.MappableTile)
res1: Class[_ >: geotrellis.raster.Tile] = null
res2: Array[java.lang.reflect.Type] = Array(interface geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>)
Simeon H.K. Fitch
@metasim
Maybe it's some kind of Scala oddity:
abstract class Foo {
  val blah: String = "foo"
}

trait Bar extends Foo {
  val merf: String = "bar"
}

classOf[Foo].isAssignableFrom(classOf[Bar])
...
res3: Boolean = false
Simeon H.K. Fitch
@metasim
@echeipesh I think this is some fundamental aspect of Scala I didn't know existed. Taking the question to Scala Users.
Grigory
@pomadchin
@metasim I think it is similar to mixins restrictions behavior
Simeon H.K. Fitch
@metasim
Not sure this is my issue....
The issue is that if one uses reflection to instantiate a class with ctor parameters (t: CellGrid[Int], e: Extent) and pass it (t: Tile, e: Extent), it will fail.
Grigory
@pomadchin
ah I was talking about how traits can extend classes
even not abstract
damn sounds like a bug in our hierarchy
Simeon H.K. Fitch
@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)
Grigory
@pomadchin
yea I see
Simeon H.K. Fitch
@metasim
The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:
scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
Grigory
@pomadchin
yea I got it why you need it :/
Simeon H.K. Fitch
@metasim
Not really a bug in your hierarchy.
It's that Spark SQL Encoders make a lot of assumptions.
Grigory
@pomadchin
yea Sparky reflection
@metasim
:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
will you file a GT issue? sounds like smth we need to resolve anyway
Grigory
@pomadchin
and in 2.x:
:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
15952026052
@15952026052
Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?
image.png
image.png
Grigory
@pomadchin
hey @15952026052
import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._
Simeon H.K. Fitch
@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052
@15952026052
thank u
Grigory
@pomadchin
@metasim I think we need to make a normal fix :/
will you file this issue? It is really sad that we havent noticed that
15952026052
@15952026052
image.png
How can I fix this?
val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram
val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)
Grigory
@pomadchin
@15952026052
import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._
15952026052
@15952026052
It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.
image.png
Grigory
@pomadchin
CollectTileLayerMetadata.fromRDD
15952026052
@15952026052
hello I am running an error about jackson
image.png
image.png
15952026052
@15952026052
image.png
I encountered this error when I used Geotrellis-3.0.0
image.png
This is my tif information. There are a few more TiffTags that can't be recognized by Geotrellis-2.1.0. So I used Geotrellis-3.0.0
15952026052
@15952026052
But when I processed another image, there was no error. This is the only difference between two tifs. And the following is the information of the tif without error.
image.png