Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 15:43
    philvarner closed #3345
  • 15:43
    philvarner commented #3345
  • 15:43
    philvarner commented #3345
  • 15:08

    pomadchin on master

    change all long literals to use… (compare)

  • 15:08
    pomadchin closed #3344
  • 14:54
    pomadchin commented #3345
  • 14:49
    philvarner commented #3344
  • 14:28
    pomadchin review_requested #3344
  • 14:28
    pomadchin review_requested #3345
  • 14:28
    pomadchin edited #3345
  • 14:25
    pomadchin commented #3344
  • 14:22
    pomadchin edited #3344
  • 14:22

    pomadchin on master

    update all methods to not use p… (compare)

  • 14:22
    pomadchin closed #3343
  • 14:22
    pomadchin edited #3343
  • 13:57
    philvarner opened #3345
  • 13:49
    philvarner opened #3344
  • 13:27
    philvarner ready_for_review #3343
  • 13:12
    philvarner opened #3343
  • 01:34

    pomadchin on master

    replace all unicode arrow opera… (compare)

iceland1906
@iceland1906
the background grayscale image is the layerRDD, each little gray box is a tile
and two green polygons consists the multipolygon
those oranges boxes are some of the tiles returned by the filter which are contained in the top polygon, the problem is none is returned from the bottom one
I would expect thousands of tiles contained in the bottom polygon as well
Grigory
@pomadchin
hm. can you create a compiling and independend from spark example?
I mean walk through all shapePolygon.contains(tileExtentPolygon) and check that tileExtentPolygon is always inside
iceland1906
@iceland1906
you mean check the tiles inside the bottom polygon to see if the tileExtentPolygon is actually inside?
Grigory
@pomadchin
yes. check the actual values that are passed into function
mb there is a mistake somewhere ~ like in metadata or somewhere else
iceland1906
@iceland1906
sure, will get back to you soon
John Smith
@GintokiYs
How can I handle a 13G tif file with Geotrellis?
I cut the 13G tif file into multiple small tif and put it in a directory. When I use Geotrellis to read this directory, I get an error.
image.png
image.png
Grigory
@pomadchin

Just an announcement: we released GeoTrellis 3.0 that depends on Spark 2.4 and has a new RasterSources API released. Also we officially have GDAL package back! https://github.com/locationtech/geotrellis/releases/tag/v3.0.0 any feedback is appreciated :tada:

It is already publish on maven central and locationtech repos

John Smith
@GintokiYs
image.png
Grigory
@pomadchin
yo @15952026052 this bug is fixed in 3.0 (locationtech/geotrellis#3088)
John Smith
@GintokiYs
Can Geotrellis-3.0 be used now?
Grigory
@pomadchin
@15952026052 yes, it is released
John Smith
@GintokiYs
My spark version is 2.3.1. Must I have spark-2.4?
iceland1906
@iceland1906
@pomadchin checked two tiles inside the bottom polygon, printed their coordinates, both should be inside but the contain function return false
multipoly2.png
metricLayerRdd.filter(
      rddKeyVal => {
        val key = rddKeyVal._1
        val tileExtentPolygon = key.extent(params.metadata.layout).jtsGeom
        val contained = shapePolygon.contains(tileExtentPolygon)
        if(tileExtentPolygon.contains(pnt.jtsGeom)) {
          println(tileExtentPolygon)
          println(shapePolygon)
          println(contained)
          System.exit(1)
        }
        contained
      })
Grigory
@pomadchin
can you throw here polygons that should be inside?
as geojsons
iceland1906
@iceland1906
sure
Grigory
@pomadchin
println(tileExtentPolygon.asJson); println(shapePolygon.asJson)
^ I think (don't really remember the API but can look into it in a while if that would be a problem)
Simeon H.K. Fitch
@metasim
Congratulations to all Geotrellis contributors :clap: :clap: :clap: . 3.0 is a massive leap forward, particularly in fundamental module architecture, providing a fantastic foundation for the future. A substantial achievement in which you should all be very proud. :smile: :smile: :smile:
David Landry
@davidlandry93
Hello all, congrats on 3.0
I have an issue I wonder if you can help me with
I keep runnning in a StackOverflow when trying to reproject and RDD to WebMercator
Exception in thread "main" java.lang.StackOverflowError
        at org.locationtech.jts.geom.CoordinateArrays.dimension(CoordinateArrays.java:44)
        at org.locationtech.jts.geom.impl.CoordinateArraySequence.<init>(CoordinateArraySequence.java:65)
        at org.locationtech.jts.geom.impl.CoordinateArraySequenceFactory.create(CoordinateArraySequenceFactory.java:55)
        at org.locationtech.jts.geom.GeometryFactory.createPoint(GeometryFactory.java:267)
        at geotrellis.vector.Point$.apply(Point.scala:25)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:87)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
        at geotrellis.vector.reproject.Reproject$.refine$1(Reproject.scala:97)
# Continued...
The offending call
  def assimilate(path: String, catalog: String)(implicit sc: SparkContext): Unit = {
    val scheme = ZoomedLayoutScheme(LatLng, tileSize = 256)

    val (zoom, rdd) = netcdfToRDD(path, scheme)
    val reprojected = rdd.reproject(WebMercator, scheme, Bilinear)

    val attributeStore = new FileAttributeStore(catalog)
    val writer = new FileLayerWriter(attributeStore, catalog)
    writer.write(LayerId("precipitation", zoom), rdd, ZCurveKeyIndexMethod.byHour())
  }
Are you guys familiar with this problem? Otherwise I'll try to send you a snippet to reproduce
iceland1906
@iceland1906

@pomadchin here are the shapePolygon and the tileExtent that is inside the polygon

    metricLayerRdd.filter(  rddKeyVal => {
        val key = rddKeyVal._1
        val tileExtentPolygon = key.extent(params.metadata.layout)
        val contained = shapePolygon.contains(tileExtentPolygon)
        if(tileExtentPolygon.contains(pnt.jtsGeom)) {
          println(tileExtentPolygon.toGeoJson())
          println(shapePolygon.toGeoJson())
          println(contained)
          System.exit(1)
        }
        contained
      })

output

iceland1906
@iceland1906

{"type":"Polygon","coordinates":[[[301599.466697037,4932229.096487437],[301599.466697037,4932230.096487457],[301600.46669720486,4932230.096487457],[301600.46669720486,4932229.096487437],[301599.466697037,4932229.096487437]]]}
{"type":"MultiPolygon","coordinates":[[[[301570.4514820544,4932330.550698328],[301608.4739502942,4932329.050574216],[301615.7797098104,4932283.3908159025],[301567.98736547644,4932285.527373691],[301570.4514820544,4932330.550698328]],[[301568.33936377685,4932265.41646293],[301616.3991255134,4932264.116287277],[301607.37711448024,4932221.9959483575],[301576.71921868704,4932223.2709015105],[301568.33936377685,4932265.41646293]]]]}
false
Grigory
@pomadchin
cool let me see
thanks @metasim :tada:
@davidlandry93 yes it is a known bug, first of all try to update up to 3.0 it will write you a better exception :D but it looks like your initial CRS of the rasters you’re reprojecting is in incorrect CRS
for instance you have an extent is some UTM zone and it is in RDD with a LatLng CRS and you’re trying to reproject it into webmercator
@davidlandry93 locationtech/geotrellis#3083 and the related issue with a similar problem you had locationtech/geotrellis#3023
David Landry
@davidlandry93
@pomadchin I'll update to 3.0. I made the input RDD myself so I might have gotten mixed up in the CRS for sure.
My extent is valid to latlng though it you compare to this https://github.com/locationtech/geotrellis/issues/3023#issuecomment-534733560
Grigory
@pomadchin
@iceland1906 what is the CRS of these polygons?
David Landry
@davidlandry93
Extent(210.0, 40.0, 309.8999938964844, 69.94000244140625)
Grigory
@pomadchin
@davidlandry93 210 and 309 are problematic here O:
David Landry
@davidlandry93
Extents are lat lon lat lon?
Grigory
@pomadchin
in LatLng longs are in [-180, 180] & lats are in [-90, 90] degree ranges
David Landry
@davidlandry93
Ohhhhhh
iceland1906
@iceland1906
@pomadchin EPSG:32616
Grigory
@pomadchin
@iceland1906 ha this looks like a bug(?)
Will look closer into it today
iceland1906
@iceland1906
thanks @pomadchin
Grigory
@pomadchin
what is tileExtentPolygon.contains(pnt.jtsGeom)?