Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 15 22:29
    pomadchin unassigned #3388
  • Jun 15 07:10
    qw845602 commented #3465
  • Jun 15 06:49
    qw845602 commented #3465
  • Jun 15 06:47
    qw845602 commented #3465
  • Jun 02 01:04

    pomadchin on npm_and_yarn

    (compare)

  • Jun 02 01:04

    pomadchin on master

    Bump eventsource from 1.0.7 to … (compare)

  • Jun 02 01:04
    pomadchin closed #3466
  • Jun 01 20:52
    dependabot[bot] labeled #3466
  • Jun 01 20:52
    dependabot[bot] opened #3466
  • Jun 01 20:52

    dependabot[bot] on npm_and_yarn

    Bump eventsource from 1.0.7 to … (compare)

  • May 27 12:47
    pomadchin synchronize #3435
  • May 22 01:34
    pomadchin commented #3465
  • May 22 01:33
    pomadchin labeled #3465
  • May 21 16:17
    pomadchin commented #3465
  • May 21 16:10
    qw845602 commented #3465
  • May 21 16:07
    qw845602 commented #3465
  • May 21 15:59
    qw845602 commented #3465
  • May 21 15:57
    qw845602 commented #3465
  • May 21 15:56
    qw845602 commented #3465
  • May 21 15:56
    pomadchin commented #3465
Mr.Gordon
@GofferdoXu
Hey @pomadchin I want to run the ogc-example project, https://github.com/geotrellis/geotrellis-server/blob/main/ogc-example,How to write url?
Max
@Max-AR
@pomadchin Using graalvm in JIT mode (not a native binary) gave about a 7-10% boost in perf, but I wasn't able to figure out how to get a native compile working to really test the differences between java byte code and graalvms binary. Do you think it would be possible to do a native-image compile of geotrellis? (not using spark that is)
Grigory
@pomadchin
Hey @Max-AR I think I mentioned somehwere above, I don’t think it is possible right now to build native image: spark has complicated dependencies and java reflection calls, uses SPI etc
the same is applicable to some GeoTrellis dependencies (like geotools) and almost every GeoTrellis package: cassandra backend, raster package - uses SPI, etc
but what errors do you have?
I think i’ve seen before a docker image with Spark
Max
@Max-AR
Yeah do you did, that is all good. Basically just having problems finding libraries at runtime. I didn't dig too much into it cause it felt like a complete rabbit hole of complications haha, I'd need to be across the geotrellis codebase a lot better to resolve those problems, i think.
Grigory
@pomadchin
Yea; I think it is impossible
the whole point of the native image is to preload classes at the build time
Spark relies on the dynamic classload
Max
@Max-AR
Yeah exactly. I was hoping that since spark/geotrellis-spark isn't a dependency in the project I am working on right now i coud skirt the issue :D
But all good
Grigory
@pomadchin
well you can try the raster and vector packages only; and without SPI usage
:+1: good plan
btw if you will endup looking into it more dont heistate to drop stacktraces
I’m also interested in the native image so mb we can craft smth together :rocket:
hey @GofferdoXu
$ sbt
$ project ogc-example
$ run http://localhost:port/
@Max-AR oooooh one thing
an issue can be related to jts
Max
@Max-AR

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

Grigory
@pomadchin
:shrug: still worth trying you don’t really need any reflection hopefully

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

makes sense +

nice nested quotes
:D
Max
@Max-AR
:D
Grigory
@pomadchin
how deep we can go?
Max
@Max-AR

raster and vector packages only; and without SPI usage

Sounds interesting, maybe I can cut down the geotrellis deps a bit more and do a minimal install

makes sense +
nice nested quotes
:D

A long way haha

Thanks again for the gudience, I'll let you know how i go
Grigory
@pomadchin
leo.jpg

Thanks again for the gudience, I'll let you know how i go

:+1: :rocket:

James Hughes
@jnh5y
@pomadchin what issues do you think JTS would cause with GraalVM?
Grigory
@pomadchin
@jnh5y not with GraalVM but with its Native Image https://www.graalvm.org/reference-manual/native-image/Limitations/
I think JTS uses some reflection; but mb it will 'just work'
Mr.Gordon
@GofferdoXu

Hi I want to read the RDD by time. This is my code

val attributeStore = FileAttributeStore("/Users/xugaofeng/tif/LC08TIME/123")
    val layerReader = FileLayerReader(attributeStore)
    val queryResult: TileLayerRDD[SpaceTimeKey] = layerReader.query[SpaceTimeKey, Tile, TileLayerMetadata[SpaceTimeKey]](LayerId("TESTTIME", 2))
      .where(Between(ZonedDateTime.of(2018, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2018, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result
    val maximumTemperature: RDD[(SpatialKey, Tile)] = queryResult.map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.localMax(_))
    val tile = maximumTemperature.stitch()
    val png = tile.renderPng(colorMapPm25)
    png.write("/Users/xugaofeng/tif/LC08TIME/123/time.png")

Error message

Exception in thread "dispatcher-event-loop-1" java.lang.StackOverflowError
    at com.esotericsoftware.kryo.util.ObjectMap.get(ObjectMap.java:351)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.getRegistration(DefaultClassResolver.java:79)
    at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
    at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:75)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
Grigory
@pomadchin
hey @GofferdoXu what GT version is that and what spark version?
is it the full stack trace?
Mr.Gordon
@GofferdoXu
@pomadchin It seems to be a problem with the temporal_resolution parameter. The parameter value is changed to 86400000, and the error will no longer occur.
Grigory
@pomadchin
Oh mb that was some runtime exception which has not been properly handled by Spark and exceptions are non serializable
Mr.Gordon
@GofferdoXu
Hey @pomadchin How to convert RDD to Tile? I know stitch at the moment.
Grigory
@pomadchin

@GofferdoXu depends on what you need

stitch stitches everything into a single raster on a driver => a pretty expensive operation

Mr.Gordon
@GofferdoXu
Is there any other way? @pomadchin
Grigory
@pomadchin
@GofferdoXu what way you need?
you can save them as chips without stitching
Mr.Gordon
@GofferdoXu
This is my code
val tile = value
      .where(Intersects(extent.center))
      .where(Between(ZonedDateTime.of(2017, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2021, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result.sortByKey(false).map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.merge(_)).stitch()
    val png = tile.renderPng()
    png
@pomadchin I want to generate png in real time and display it on the map
Grigory
@pomadchin
so usually; you need to use Value reader in this case
and if you have the TMS map than each key would correspond to a tile on a map
Mr.Gordon
@GofferdoXu
But I want to perform a query operation😅
Grigory
@pomadchin
in this case that is the only way
if you’re sure that the result will always be small you can use the CollectionLayerReader for the temporal query to make it faster; the API would be the same but you wont need Spark
Mr.Gordon
@GofferdoXu
@pomadchin Thank you.