geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Grigory

@pomadchin

ah I was talking about how traits can extend classes

even not abstract

damn sounds like a bug in our hierarchy

Simeon H.K. Fitch

@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)

Grigory

@pomadchin

yea I see

Simeon H.K. Fitch

@metasim

The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:

scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"

Grigory

@pomadchin

yea I got it why you need it :/

Simeon H.K. Fitch

@metasim

Not really a bug in your hierarchy.

It's that Spark SQL Encoders make a lot of assumptions.

Grigory

@pomadchin

yea Sparky reflection

@metasim

:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

will you file a GT issue? sounds like smth we need to resolve anyway

Grigory

@pomadchin

and in 2.x:

:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>

15952026052

@15952026052

Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?

Grigory

@pomadchin

hey @15952026052

import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._

Simeon H.K. Fitch

@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052

@15952026052

thank u

Grigory

@pomadchin

@metasim I think we need to make a normal fix :/

will you file this issue? It is really sad that we havent noticed that

15952026052

@15952026052

How can I fix this?

val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram

val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)

Grigory

@pomadchin

@15952026052

import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._

15952026052

@15952026052

It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.

Grigory

@pomadchin

CollectTileLayerMetadata.fromRDD

15952026052

@15952026052

hello I am running an error about jackson

Grigory

@pomadchin

@15952026052 https://github.com/locationtech/geotrellis/blob/master/project/Settings.scala#L315-L327

15952026052

@15952026052

I encountered this error when I used Geotrellis-3.0.0

This is my tif information. There are a few more TiffTags that can't be recognized by Geotrellis-2.1.0. So I used Geotrellis-3.0.0

15952026052

@15952026052

But when I processed another image, there was no error. This is the only difference between two tifs. And the following is the information of the tif without error.

15952026052

@15952026052

geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: No StripByteCount information.
    at geotrellis.raster.io.geotiff.tags.TiffTags.segmentCount(TiffTags.scala:556)
    at geotrellis.raster.io.geotiff.compression.Decompressor$.apply(Decompressor.scala:96)
    at 
geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.getGeoTiffInfo$1(GeoTiffInfo.scala:179)

Grigory

@pomadchin

Hey, this is probably a bug, is it a public data? I can create a unit test and we’ll fix it in our next release

Grigory

@pomadchin

the difference is not in these tags you pointed to

it looks like the issue is that we cant read tiffs with the stripe size infeormation (in case it is a striped tiff)

Grigory

@pomadchin

Without** (typo, wrote from phone)

Simeon H.K. Fitch

@metasim

@pomadchin @echeipesh Let me know if my writupe of #3134 is unclear, and if you want more out-of-band discussion of it and what our options are around it.

Grigory

@pomadchin

@metasim :100:

Eugene Cheipesh

@echeipesh

@metasim that's super clear. Seems like there are only two options? either make abstract class Tile or push Grid and CellGrid back into trait.

Grigory

@pomadchin

the second option will definitely work

lets try the first one and see would it break smth?

Grigory

@pomadchin

Tile, ArrayTile, MultibandTile to abstract classes

ArrayMultibandTile is already a class

Where communities thrive

People

Repo info

Activity