geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Simeon H.K. Fitch

@metasim

Congratulations to all Geotrellis contributors :clap: :clap: :clap: . 3.0 is a massive leap forward, particularly in fundamental module architecture, providing a fantastic foundation for the future. A substantial achievement in which you should all be very proud. :smile: :smile: :smile:

David Landry

@davidlandry93

Hello all, congrats on 3.0
I have an issue I wonder if you can help me with
I keep runnning in a StackOverflow when trying to reproject and RDD to WebMercator

Exception in thread "main" java.lang.StackOverflowError
        at org.locationtech.jts.geom.CoordinateArrays.dimension(CoordinateArrays.java:44)
        at org.locationtech.jts.geom.impl.CoordinateArraySequence.<init>(CoordinateArraySequence.java:65)
        at org.locationtech.jts.geom.impl.CoordinateArraySequenceFactory.create(CoordinateArraySequenceFactory.java:55)
        at org.locationtech.jts.geom.GeometryFactory.createPoint(GeometryFactory.java:267)
        at geotrellis.vector.Point$.apply(Point.scala:25)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:87)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
# Continued...

The offending call

  def assimilate(path: String, catalog: String)(implicit sc: SparkContext): Unit = {
    val scheme = ZoomedLayoutScheme(LatLng, tileSize = 256)

    val (zoom, rdd) = netcdfToRDD(path, scheme)
    val reprojected = rdd.reproject(WebMercator, scheme, Bilinear)

    val attributeStore = new FileAttributeStore(catalog)
    val writer = new FileLayerWriter(attributeStore, catalog)
    writer.write(LayerId("precipitation", zoom), rdd, ZCurveKeyIndexMethod.byHour())
  }

Are you guys familiar with this problem? Otherwise I'll try to send you a snippet to reproduce

iceland1906

@iceland1906

@pomadchin here are the shapePolygon and the tileExtent that is inside the polygon

    metricLayerRdd.filter(  rddKeyVal => {
        val key = rddKeyVal._1
        val tileExtentPolygon = key.extent(params.metadata.layout)
        val contained = shapePolygon.contains(tileExtentPolygon)
        if(tileExtentPolygon.contains(pnt.jtsGeom)) {
          println(tileExtentPolygon.toGeoJson())
          println(shapePolygon.toGeoJson())
          println(contained)
          System.exit(1)
        }
        contained
      })

output

iceland1906

@iceland1906


{"type":"Polygon","coordinates":[[[301599.466697037,4932229.096487437],[301599.466697037,4932230.096487457],[301600.46669720486,4932230.096487457],[301600.46669720486,4932229.096487437],[301599.466697037,4932229.096487437]]]}
{"type":"MultiPolygon","coordinates":[[[[301570.4514820544,4932330.550698328],[301608.4739502942,4932329.050574216],[301615.7797098104,4932283.3908159025],[301567.98736547644,4932285.527373691],[301570.4514820544,4932330.550698328]],[[301568.33936377685,4932265.41646293],[301616.3991255134,4932264.116287277],[301607.37711448024,4932221.9959483575],[301576.71921868704,4932223.2709015105],[301568.33936377685,4932265.41646293]]]]}
false

Grigory

@pomadchin

cool let me see

thanks @metasim :tada:

@davidlandry93 yes it is a known bug, first of all try to update up to 3.0 it will write you a better exception :D but it looks like your initial CRS of the rasters you’re reprojecting is in incorrect CRS

for instance you have an extent is some UTM zone and it is in RDD with a LatLng CRS and you’re trying to reproject it into webmercator

@davidlandry93 locationtech/geotrellis#3083 and the related issue with a similar problem you had locationtech/geotrellis#3023

David Landry

@davidlandry93

@pomadchin I'll update to 3.0. I made the input RDD myself so I might have gotten mixed up in the CRS for sure.

My extent is valid to latlng though it you compare to this https://github.com/locationtech/geotrellis/issues/3023#issuecomment-534733560

Grigory

@pomadchin

@iceland1906 what is the CRS of these polygons?

David Landry

@davidlandry93

Extent(210.0, 40.0, 309.8999938964844, 69.94000244140625)

Grigory

@pomadchin

@davidlandry93 210 and 309 are problematic here O:

David Landry

@davidlandry93

Extents are lat lon lat lon?

Grigory

@pomadchin

in LatLng longs are in [-180, 180] & lats are in [-90, 90] degree ranges

David Landry

@davidlandry93

Ohhhhhh

iceland1906

@iceland1906

@pomadchin EPSG:32616

Grigory

@pomadchin

@iceland1906 ha this looks like a bug(?)

Will look closer into it today

iceland1906

@iceland1906

thanks @pomadchin

Grigory

@pomadchin

what is tileExtentPolygon.contains(pnt.jtsGeom)?

in your code above

and what is the type of shapePolygon?

ah nvm I think I fugred out how to reproduce ghe behavior you had

will create a unit test for it

iceland1906

@iceland1906

@pomadchin tileExtentPolygon.contains(pnt.jtsGeom) is just a condition to only show one tile in the target polygon, for demo purpose, not related with the issue

shapePolygon is geotrellis.vector.MultiPolygon type

Ricardo Yrupailla Meza

@stg101

Hi everyone, congratulations on 3.0 !
How can I save

a geotrellis.spark.TileLayerRDD[geotrellis.layer.SpatialKey] as a single tif file

btw I am starting to use geotrellis, so excuse me if the question is too simple.

iceland1906

@iceland1906

@stg101 try Section "Stiching Tiles into a single GeoTiff" at this link https://geotrellis.readthedocs.io/en/latest/guide/spark.html

Ricardo Yrupailla Meza

@stg101

Thanks I'm going to try that

I got the following error

[error]  No implicit view available from geotrellis.layer.SpatialKey => geotrellis.spark.tiling.TilerKeyMethods[geotrellis.layer.SpatialKey,geotrellis.layer.SpatialKey].
[error]       rdd.collectMetadata[SpatialKey](layoutScheme)
[error]

In this line

    val (_: Int, metadata: TileLayerMetadata[SpatialKey]) =
      rdd.collectMetadata[SpatialKey](layoutScheme)

Grigory

@pomadchin

hey @stg101 yea, but it depends on what do you want: you can stitch TileLayerRDD into a single raster and persist as a tiff (but it would trigger the entire rdd reduce on a driver) or you can group it by and save by chunks

what is the type of your RDD? during the collect metadata step rdd.collectMetadata

Ricardo Yrupailla Meza

@stg101

I would like to stitch the TileLayerRDD into a single raster

The type is geotrellis.spark.TileLayerRDD[geotrellis.layer.SpatialKey]

Grigory

@pomadchin

Yea so collectMetadata works only for RDD[ProjectedExtent, T]

you already have all metadata in rdd.metadata

the only thing you need to do is to call rdd.stitch and that is it

val rdd: TileLayerRDD[SpatialKey] = ???
val raster: Raster[Tile] = rdd.stitch
GeoTiff(raster, rdd.metadata.crs).write(“path/to/output.tiff”)

Ricardo Yrupailla Meza

@stg101

Thank you so much !

Grigory

@pomadchin

hey @iceland1906 how do you get your MultiPolygon? from geoJson?

iceland1906

@iceland1906

yes

polygonType match {
      case "Polygon" =>
        val polygon = jsonString.parseGeoJson[Polygon]
        val reprojectedPolygon = polygon.reproject(CRS.fromEpsgCode(polygonEpsg), CRS.fromEpsgCode(epsgCode))
        MultiPolygon(Array(reprojectedPolygon))
      case "MultiPolygon" =>
        val multiPolygon = jsonString.parseGeoJson[MultiPolygon]
        multiPolygon.reproject(CRS.fromEpsgCode(polygonEpsg), CRS.fromEpsgCode(epsgCode))
    }

Where communities thrive

People

Repo info

Activity