Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 21 18:14
    pomadchin edited #3133
  • Oct 21 18:05
    pomadchin edited #3133
  • Oct 21 17:57
    echeipesh synchronize #3132
  • Oct 21 17:33
    pomadchin opened #3133
  • Oct 21 17:33
    pomadchin labeled #3133
  • Oct 21 16:39
    echeipesh labeled #3132
  • Oct 21 16:39
    echeipesh opened #3132
  • Oct 21 16:39
    echeipesh labeled #3132
  • Oct 21 14:53
    echeipesh unassigned #3127
  • Oct 21 14:52
    echeipesh assigned #3127
  • Oct 21 14:52
    echeipesh unassigned #3127
  • Oct 21 14:52
    echeipesh assigned #3127
  • Oct 21 14:45
    echeipesh assigned #3131
  • Oct 21 14:42
    echeipesh unlabeled #3123
  • Oct 19 19:05
    echeipesh labeled #3131
  • Oct 19 19:05
    echeipesh opened #3131
  • Oct 18 21:53
    echeipesh edited #2932
  • Oct 18 21:52
    echeipesh edited #2932
  • Oct 18 19:47
    echeipesh milestoned #3100
  • Oct 18 19:41
    echeipesh demilestoned #2981
Eugene Cheipesh
@echeipesh
classOf wouldn't hold any information about the type parameter though
Simeon H.K. Fitch
@metasim
I just noticed that CellGrid is an abstract class, and Tile extents it... didn't even know that was possible!
Ok, this is super spooky.... CellGrid doesn't show up through reflection!:
scala> classOf[Tile].getInterfaces
scala> classOf[Tile].getSuperclass
scala> classOf[Tile].getGenericInterfaces

res0: Array[Class[_]] = Array(interface geotrellis.raster.IterableTile, interface geotrellis.raster.MappableTile)
res1: Class[_ >: geotrellis.raster.Tile] = null
res2: Array[java.lang.reflect.Type] = Array(interface geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>)
Simeon H.K. Fitch
@metasim
Maybe it's some kind of Scala oddity:
abstract class Foo {
  val blah: String = "foo"
}

trait Bar extends Foo {
  val merf: String = "bar"
}

classOf[Foo].isAssignableFrom(classOf[Bar])
...
res3: Boolean = false
Simeon H.K. Fitch
@metasim
@echeipesh I think this is some fundamental aspect of Scala I didn't know existed. Taking the question to Scala Users.
Grigory
@pomadchin
@metasim I think it is similar to mixins restrictions behavior
Simeon H.K. Fitch
@metasim
Not sure this is my issue....
The issue is that if one uses reflection to instantiate a class with ctor parameters (t: CellGrid[Int], e: Extent) and pass it (t: Tile, e: Extent), it will fail.
Grigory
@pomadchin
ah I was talking about how traits can extend classes
even not abstract
damn sounds like a bug in our hierarchy
Simeon H.K. Fitch
@metasim

Yeh, I'm more hung up on the fact that the relationship doesn't exist in the JVM. Here's the trigger:

scala> val rc = classOf[Raster[Tile]]
rc: Class[geotrellis.raster.Raster[geotrellis.raster.Tile]] = class geotrellis.raster.Raster

scala> rc.getConstructors
res0: Array[java.lang.reflect.Constructor[_]] = Array(public geotrellis.raster.Raster(geotrellis.raster.CellGrid,geotrellis.vector.Extent))

scala> rc.getConstructor(classOf[Tile], classOf[Extent])
java.lang.NoSuchMethodException: geotrellis.raster.Raster.<init>(geotrellis.raster.Tile, geotrellis.vector.Extent)

However, it's clearly in the compile-time typing:

scala> import scala.reflect.runtime.universe._
scala> typeOf[Tile].baseClasses

res0: List[reflect.runtime.universe.Symbol] = List(trait Tile, trait MappableTile, trait MacroMappableTile, trait IterableTile, trait MacroIterableTile, class CellGrid, class Grid, trait Serializable, trait Serializable, class Object, class Any)
Grigory
@pomadchin
yea I see
Simeon H.K. Fitch
@metasim
The reason this is relevant for me has to do with Spark SQL's encoding of product types (e.g. case classes) expects a ctor to exist:
scala> rf.printSchema()
raster
 |-- tile: tile (nullable = true)
 |-- extent: struct (nullable = true)
 |    |-- xmin: double (nullable = false)
 |    |-- ymin: double (nullable = false)
 |    |-- xmax: double (nullable = false)
 |    |-- ymax: double (nullable = false)

scala> rf .select(col("raster").as[Raster[Tile]])

ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 102, Column 11: No applicable constructor/method found for actual parameters "geotrellis.raster.Tile, geotrellis.vector.Extent"; candidates are: "geotrellis.raster.Raster(geotrellis.raster.CellGrid, geotrellis.vector.Extent)"
Grigory
@pomadchin
yea I got it why you need it :/
Simeon H.K. Fitch
@metasim
Not really a bug in your hierarchy.
It's that Spark SQL Encoders make a lot of assumptions.
Grigory
@pomadchin
yea Sparky reflection
@metasim
:javap geotrellis.raster.Tile
  Size 5432 bytes
  MD5 checksum 130428f0bb0a80b0748ad561f426f1ab
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
will you file a GT issue? sounds like smth we need to resolve anyway
Grigory
@pomadchin
and in 2.x:
:javap Tile
  Size 15801 bytes
  MD5 checksum 603f5495ea4c630a86414555994a052a
  Compiled from "Tile.scala"
public interface geotrellis.raster.Tile extends geotrellis.raster.CellGrid, geotrellis.raster.IterableTile, geotrellis.raster.MappableTile<geotrellis.raster.Tile>
15952026052
@15952026052
Hello When I upgraded to 3.0.0, the original guide package error occurred.
How can I fix it?
image.png
image.png
Grigory
@pomadchin
hey @15952026052
import geotrellis.store._
import geotrellis.store.index._
import geotrellis.store.file._
import geotrellis.store.hadoop._
Simeon H.K. Fitch
@metasim

@pomadchin This might be a fix (for me at least):

case class Raster[+T <: CellGrid[Int]](tile: T, extent: Extent) ...  {
  def this(tile: Tile, extent: Extent) = this(tile.asInstanceOf[T], extent)
...
}

Need to test, but how yucky would you find this?

15952026052
@15952026052
thank u
Grigory
@pomadchin
@metasim I think we need to make a normal fix :/
will you file this issue? It is really sad that we havent noticed that
15952026052
@15952026052
image.png
How can I fix this?
val inputRdd: RDD[(ProjectedExtent, MultibandTile)] =
sc.hadoopMultibandGeoTiffRDD(inputPath).mapValues(m => m.withNoData(Option(0)))
val his: Array[Histogram[Double]] = inputRdd.histogram
val (_, rasterMetaData) =
  TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))


val tiled: RDD[(SpatialKey, MultibandTile)] =
  inputRdd
    .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
    .repartition(200)
Grigory
@pomadchin
@15952026052
import geotrellis.spark._
import geotrellis.spark.store._
import geotrellis.spark.store.hadoop._
15952026052
@15952026052
It works and fixes an error but how do I get the parameter rasterMetaData Geotrellis-3.0.0 without this method.
image.png
Grigory
@pomadchin
CollectTileLayerMetadata.fromRDD
15952026052
@15952026052
hello I am running an error about jackson
image.png
image.png
15952026052
@15952026052
image.png
I encountered this error when I used Geotrellis-3.0.0
image.png
This is my tif information. There are a few more TiffTags that can't be recognized by Geotrellis-2.1.0. So I used Geotrellis-3.0.0
15952026052
@15952026052
But when I processed another image, there was no error. This is the only difference between two tifs. And the following is the information of the tif without error.
image.png
15952026052
@15952026052
geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: No StripByteCount information.
    at geotrellis.raster.io.geotiff.tags.TiffTags.segmentCount(TiffTags.scala:556)
    at geotrellis.raster.io.geotiff.compression.Decompressor$.apply(Decompressor.scala:96)
    at 
geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.getGeoTiffInfo$1(GeoTiffInfo.scala:179)
Grigory
@pomadchin
Hey, this is probably a bug, is it a public data? I can create a unit test and we’ll fix it in our next release
Grigory
@pomadchin
the difference is not in these tags you pointed to
it looks like the issue is that we cant read tiffs with the stripe size infeormation (in case it is a striped tiff)