geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Eugene Cheipesh

@echeipesh

classOf wouldn't hold any information about the type parameter though

Simeon H.K. Fitch

@metasim

I just noticed that CellGrid is an abstract class, and Tile extents it... didn't even know that was possible!

Ok, this is super spooky.... CellGrid doesn't show up through reflection!:

scala> classOf[Tile].getInterfaces
scala> classOf[Tile].getSuperclass
scala> classOf[Tile].getGenericInterfaces

res0: Array[Class[_]] = Array(interface geotrellis.raster.IterableTile, interface geotrellis.raster.MappableTile)
res1: Class[_ >: geotrellis.raster.Tile] = null
res2: Array[java.lang.reflect.Type] = Array(interface geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>)

Simeon H.K. Fitch

@metasim

Maybe it's some kind of Scala oddity:

abstract class Foo {
  val blah: String = "foo"
}

trait Bar extends Foo {
  val merf: String = "bar"
}

classOf[Foo].isAssignableFrom(classOf[Bar])
...
res3: Boolean = false

Simeon H.K. Fitch

@metasim

@echeipesh I think this is some fundamental aspect of Scala I didn't know existed. Taking the question to Scala Users.

Grigory

@pomadchin

@metasim I think it is similar to mixins restrictions behavior

Grigory

@pomadchin

@metasim https://alvinalexander.com/scala/scala-trait-examples#limiting-which-classes-can-use-a-trait

Simeon H.K. Fitch

@metasim

Not sure this is my issue....
The issue is that if one uses reflection to instantiate a class with ctor parameters (t: CellGrid[Int], e: Extent) and pass it (t: Tile, e: Extent), it will fail.

Grigory

@pomadchin

ah I was talking about how traits can extend classes

even not abstract

damn sounds like a bug in our hierarchy

Simeon H.K. Fitch

@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)

Grigory

@pomadchin

yea I see

Simeon H.K. Fitch

@metasim

The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:

scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"

Grigory

@pomadchin

yea I got it why you need it :/

Simeon H.K. Fitch

@metasim

Not really a bug in your hierarchy.

It's that Spark SQL Encoders make a lot of assumptions.

Grigory

@pomadchin

yea Sparky reflection

@metasim

:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

will you file a GT issue? sounds like smth we need to resolve anyway

Grigory

@pomadchin

and in 2.x:

:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

15952026052

@15952026052

Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?

Grigory

@pomadchin

hey @15952026052

import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._

Simeon H.K. Fitch

@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052

@15952026052

thank u

Grigory

@pomadchin

@metasim I think we need to make a normal fix :/

will you file this issue? It is really sad that we havent noticed that

15952026052

@15952026052

How can I fix this?

val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram

val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)

Grigory

@pomadchin

@15952026052

import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._

15952026052

@15952026052

It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.

Grigory

@pomadchin

CollectTileLayerMetadata.fromRDD

15952026052

@15952026052

hello I am running an error about jackson

Grigory

@pomadchin

@15952026052 https://github.com/locationtech/geotrellis/blob/master/project/Settings.scala#L315-L327

15952026052

@15952026052

I encountered this error when I used Geotrellis-3.0.0

This is my tif information. There are a few more TiffTags that can't be recognized by Geotrellis-2.1.0. So I used Geotrellis-3.0.0

15952026052

@15952026052

But when I processed another image, there was no error. This is the only difference between two tifs. And the following is the information of the tif without error.

15952026052

@15952026052

geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: No StripByteCount information.
    at geotrellis.raster.io.geotiff.tags.TiffTags.segmentCount(TiffTags.scala:556)
    at geotrellis.raster.io.geotiff.compression.Decompressor$.apply(Decompressor.scala:96)
    at 
geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.getGeoTiffInfo$1(GeoTiffInfo.scala:179)

Grigory

@pomadchin

Hey, this is probably a bug, is it a public data? I can create a unit test and we’ll fix it in our next release

Grigory

@pomadchin

the difference is not in these tags you pointed to

it looks like the issue is that we cant read tiffs with the stripe size infeormation (in case it is a striped tiff)

Where communities thrive

People

Repo info

Activity