geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Grigory

@pomadchin

Spark relies on the dynamic classload

Max

@Max-AR

Yeah exactly. I was hoping that since spark/geotrellis-spark isn't a dependency in the project I am working on right now i coud skirt the issue :D

But all good

Grigory

@pomadchin

well you can try the raster and vector packages only; and without SPI usage

:+1: good plan

btw if you will endup looking into it more dont heistate to drop stacktraces

I’m also interested in the native image so mb we can craft smth together :rocket:

hey @GofferdoXu

$ sbt
$ project ogc-example
$ run http://localhost:port/

@Max-AR oooooh one thing

an issue can be related to jts

Max

@Max-AR

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

Grigory

@pomadchin

:shrug: still worth trying you don’t really need any reflection hopefully

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

makes sense +

nice nested quotes

Max

@Max-AR

Grigory

@pomadchin

how deep we can go?

Max

@Max-AR

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

makes sense +
nice nested quotes
:D

A long way haha

Thanks again for the gudience, I'll let you know how i go

Grigory

@pomadchin

Thanks again for the gudience, I'll let you know how i go

:+1: :rocket:

James Hughes

@jnh5y

@pomadchin what issues do you think JTS would cause with GraalVM?

Grigory

@pomadchin

@jnh5y not with GraalVM but with its Native Image https://www.graalvm.org/reference-manual/native-image/Limitations/

I think JTS uses some reflection; but mb it will 'just work'

Mr.Gordon

@GofferdoXu

Hi I want to read the RDD by time. This is my code

val attributeStore = FileAttributeStore("/Users/xugaofeng/tif/LC08TIME/123")
    val layerReader = FileLayerReader(attributeStore)
    val queryResult: TileLayerRDD[SpaceTimeKey] = layerReader.query[SpaceTimeKey, Tile, TileLayerMetadata[SpaceTimeKey]](LayerId("TESTTIME", 2))
      .where(Between(ZonedDateTime.of(2018, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2018, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result
    val maximumTemperature: RDD[(SpatialKey, Tile)] = queryResult.map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.localMax(_))
    val tile = maximumTemperature.stitch()
    val png = tile.renderPng(colorMapPm25)
    png.write("/Users/xugaofeng/tif/LC08TIME/123/time.png")

Error message

Exception in thread "dispatcher-event-loop-1" java.lang.StackOverflowError
    at com.esotericsoftware.kryo.util.ObjectMap.get(ObjectMap.java:351)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.getRegistration(DefaultClassResolver.java:79)
    at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
    at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:75)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)

Grigory

@pomadchin

hey @GofferdoXu what GT version is that and what spark version?

is it the full stack trace?

Mr.Gordon

@GofferdoXu

@pomadchin It seems to be a problem with the temporal_resolution parameter. The parameter value is changed to 86400000, and the error will no longer occur.

Grigory

@pomadchin

Oh mb that was some runtime exception which has not been properly handled by Spark and exceptions are non serializable

Mr.Gordon

@GofferdoXu

Hey @pomadchin How to convert RDD to Tile? I know stitch at the moment.

Grigory

@pomadchin

@GofferdoXu depends on what you need

stitch stitches everything into a single raster on a driver => a pretty expensive operation

Mr.Gordon

@GofferdoXu

Is there any other way? @pomadchin

Grigory

@pomadchin

@GofferdoXu what way you need?

you can save them as chips without stitching

see https://github.com/locationtech/geotrellis/blob/master/s3-spark/src/main/scala/geotrellis/spark/store/s3/SaveToS3.scala

Mr.Gordon

@GofferdoXu

This is my code

val tile = value
      .where(Intersects(extent.center))
      .where(Between(ZonedDateTime.of(2017, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2021, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result.sortByKey(false).map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.merge(_)).stitch()
    val png = tile.renderPng()
    png

@pomadchin I want to generate png in real time and display it on the map

Grigory

@pomadchin

so usually; you need to use Value reader in this case

and if you have the TMS map than each key would correspond to a tile on a map

Mr.Gordon

@GofferdoXu

But I want to perform a query operation😅

Grigory

@pomadchin

in this case that is the only way

if you’re sure that the result will always be small you can use the CollectionLayerReader for the temporal query to make it faster; the API would be the same but you wont need Spark

Mr.Gordon

@GofferdoXu

@pomadchin Thank you.

Grigory

@pomadchin

@GofferdoXu :+1:

Mr.Gordon

@GofferdoXu

Hey @pomadchin I still have a question. I use the pipeline, and the picture that comes out looks like this, and it doesn’t look normal.

😅

Grigory

@pomadchin

hey @GofferdoXu what do you mean by “doesnt look normal” ?

how should it look like?

Mr.Gordon

@GofferdoXu

@pomadchin I think it should be a remote sensing image, not such a dotted image

Where communities thrive

People

Repo info

Activity