geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

iceland1906

@iceland1906

the background grayscale image is the layerRDD, each little gray box is a tile

and two green polygons consists the multipolygon

those oranges boxes are some of the tiles returned by the filter which are contained in the top polygon, the problem is none is returned from the bottom one

I would expect thousands of tiles contained in the bottom polygon as well

Grigory

@pomadchin

hm. can you create a compiling and independend from spark example?
I mean walk through all shapePolygon.contains(tileExtentPolygon) and check that tileExtentPolygon is always inside

iceland1906

@iceland1906

you mean check the tiles inside the bottom polygon to see if the tileExtentPolygon is actually inside?

Grigory

@pomadchin

yes. check the actual values that are passed into function

mb there is a mistake somewhere ~ like in metadata or somewhere else

iceland1906

@iceland1906

sure, will get back to you soon

John Smith

@GintokiYs

How can I handle a 13G tif file with Geotrellis?
I cut the 13G tif file into multiple small tif and put it in a directory. When I use Geotrellis to read this directory, I get an error.

Grigory

@pomadchin

Just an announcement: we released GeoTrellis 3.0 that depends on Spark 2.4 and has a new RasterSources API released. Also we officially have GDAL package back! https://github.com/locationtech/geotrellis/releases/tag/v3.0.0 any feedback is appreciated :tada:

It is already publish on maven central and locationtech repos

John Smith

@GintokiYs

Grigory

@pomadchin

yo @15952026052 this bug is fixed in 3.0 (locationtech/geotrellis#3088)

John Smith

@GintokiYs

Can Geotrellis-3.0 be used now?

Grigory

@pomadchin

@15952026052 yes, it is released

John Smith

@GintokiYs

My spark version is 2.3.1. Must I have spark-2.4?

iceland1906

@iceland1906

@pomadchin checked two tiles inside the bottom polygon, printed their coordinates, both should be inside but the contain function return false

metricLayerRdd.filter(
      rddKeyVal => {
        val key = rddKeyVal._1
        val tileExtentPolygon = key.extent(params.metadata.layout).jtsGeom
        val contained = shapePolygon.contains(tileExtentPolygon)
        if(tileExtentPolygon.contains(pnt.jtsGeom)) {
          println(tileExtentPolygon)
          println(shapePolygon)
          println(contained)
          System.exit(1)
        }
        contained
      })

Grigory

@pomadchin

can you throw here polygons that should be inside?

as geojsons

iceland1906

@iceland1906

sure

Grigory

@pomadchin

println(tileExtentPolygon.asJson); println(shapePolygon.asJson)

^ I think (don't really remember the API but can look into it in a while if that would be a problem)

Simeon H.K. Fitch

@metasim

Congratulations to all Geotrellis contributors :clap: :clap: :clap: . 3.0 is a massive leap forward, particularly in fundamental module architecture, providing a fantastic foundation for the future. A substantial achievement in which you should all be very proud. :smile: :smile: :smile:

David Landry

@davidlandry93

Hello all, congrats on 3.0
I have an issue I wonder if you can help me with
I keep runnning in a StackOverflow when trying to reproject and RDD to WebMercator

Exception in thread "main" java.lang.StackOverflowError
        at org.locationtech.jts.geom.CoordinateArrays.dimension(CoordinateArrays.java:44)
        at org.locationtech.jts.geom.impl.CoordinateArraySequence.<init>(CoordinateArraySequence.java:65)
        at org.locationtech.jts.geom.impl.CoordinateArraySequenceFactory.create(CoordinateArraySequenceFactory.java:55)
        at org.locationtech.jts.geom.GeometryFactory.createPoint(GeometryFactory.java:267)
        at geotrellis.vector.Point$.apply(Point.scala:25)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:87)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
# Continued...

The offending call

  def assimilate(path: String, catalog: String)(implicit sc: SparkContext): Unit = {
    val scheme = ZoomedLayoutScheme(LatLng, tileSize = 256)

    val (zoom, rdd) = netcdfToRDD(path, scheme)
    val reprojected = rdd.reproject(WebMercator, scheme, Bilinear)

    val attributeStore = new FileAttributeStore(catalog)
    val writer = new FileLayerWriter(attributeStore, catalog)
    writer.write(LayerId("precipitation", zoom), rdd, ZCurveKeyIndexMethod.byHour())
  }

Are you guys familiar with this problem? Otherwise I'll try to send you a snippet to reproduce

iceland1906

@iceland1906

@pomadchin here are the shapePolygon and the tileExtent that is inside the polygon

    metricLayerRdd.filter(  rddKeyVal => {
        val key = rddKeyVal._1
        val tileExtentPolygon = key.extent(params.metadata.layout)
        val contained = shapePolygon.contains(tileExtentPolygon)
        if(tileExtentPolygon.contains(pnt.jtsGeom)) {
          println(tileExtentPolygon.toGeoJson())
          println(shapePolygon.toGeoJson())
          println(contained)
          System.exit(1)
        }
        contained
      })

output

iceland1906

@iceland1906


{"type":"Polygon","coordinates":[[[301599.466697037,4932229.096487437],[301599.466697037,4932230.096487457],[301600.46669720486,4932230.096487457],[301600.46669720486,4932229.096487437],[301599.466697037,4932229.096487437]]]}
{"type":"MultiPolygon","coordinates":[[[[301570.4514820544,4932330.550698328],[301608.4739502942,4932329.050574216],[301615.7797098104,4932283.3908159025],[301567.98736547644,4932285.527373691],[301570.4514820544,4932330.550698328]],[[301568.33936377685,4932265.41646293],[301616.3991255134,4932264.116287277],[301607.37711448024,4932221.9959483575],[301576.71921868704,4932223.2709015105],[301568.33936377685,4932265.41646293]]]]}
false

Grigory

@pomadchin

cool let me see

thanks @metasim :tada:

@davidlandry93 yes it is a known bug, first of all try to update up to 3.0 it will write you a better exception :D but it looks like your initial CRS of the rasters you’re reprojecting is in incorrect CRS

for instance you have an extent is some UTM zone and it is in RDD with a LatLng CRS and you’re trying to reproject it into webmercator

@davidlandry93 locationtech/geotrellis#3083 and the related issue with a similar problem you had locationtech/geotrellis#3023

David Landry

@davidlandry93

@pomadchin I'll update to 3.0. I made the input RDD myself so I might have gotten mixed up in the CRS for sure.

My extent is valid to latlng though it you compare to this https://github.com/locationtech/geotrellis/issues/3023#issuecomment-534733560

Grigory

@pomadchin

@iceland1906 what is the CRS of these polygons?

David Landry

@davidlandry93

Extent(210.0, 40.0, 309.8999938964844, 69.94000244140625)

Grigory

@pomadchin

@davidlandry93 210 and 309 are problematic here O:

David Landry

@davidlandry93

Extents are lat lon lat lon?

Grigory

@pomadchin

in LatLng longs are in [-180, 180] & lats are in [-90, 90] degree ranges

David Landry

@davidlandry93

Ohhhhhh

iceland1906

@iceland1906

@pomadchin EPSG:32616

Grigory

@pomadchin

@iceland1906 ha this looks like a bug(?)

Will look closer into it today

iceland1906

@iceland1906

thanks @pomadchin

Where communities thrive

People

Repo info

Activity