Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 17 01:42
    pomadchin closed #3433
  • May 17 01:42
    pomadchin commented #3433
  • May 12 18:20
    axb21 commented #3464
  • May 12 18:16

    pomadchin on master

    Expose Charset in a ShapeFileRe… (compare)

  • May 12 18:16
    pomadchin closed #3464
  • May 12 18:16
    pomadchin closed #3445
  • May 12 17:30
    vlulla commented #3464
  • May 12 17:13
    pomadchin edited #3464
  • May 12 17:09
    pomadchin edited #3464
  • May 12 17:09
    pomadchin synchronize #3464
  • May 12 17:03
    vlulla synchronize #3464
  • May 12 16:33
    pomadchin edited #3464
  • May 12 15:36
    vlulla review_requested #3464
  • May 12 13:25
    vlulla commented #3464
  • May 12 13:22
    vlulla synchronize #3464
  • May 11 15:21
    pomadchin synchronize #3464
  • May 11 15:18
    vlulla commented #3464
  • May 11 15:16
    vlulla synchronize #3464
  • May 11 15:12
    pomadchin edited #3464
  • May 11 15:12
    pomadchin edited #3464
Mr.Gordon
@GofferdoXu

Hi I want to read the RDD by time. This is my code

val attributeStore = FileAttributeStore("/Users/xugaofeng/tif/LC08TIME/123")
    val layerReader = FileLayerReader(attributeStore)
    val queryResult: TileLayerRDD[SpaceTimeKey] = layerReader.query[SpaceTimeKey, Tile, TileLayerMetadata[SpaceTimeKey]](LayerId("TESTTIME", 2))
      .where(Between(ZonedDateTime.of(2018, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2018, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result
    val maximumTemperature: RDD[(SpatialKey, Tile)] = queryResult.map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.localMax(_))
    val tile = maximumTemperature.stitch()
    val png = tile.renderPng(colorMapPm25)
    png.write("/Users/xugaofeng/tif/LC08TIME/123/time.png")

Error message

Exception in thread "dispatcher-event-loop-1" java.lang.StackOverflowError
    at com.esotericsoftware.kryo.util.ObjectMap.get(ObjectMap.java:351)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.getRegistration(DefaultClassResolver.java:79)
    at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
    at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:75)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
    at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
Grigory
@pomadchin
hey @GofferdoXu what GT version is that and what spark version?
is it the full stack trace?
Mr.Gordon
@GofferdoXu
@pomadchin It seems to be a problem with the temporal_resolution parameter. The parameter value is changed to 86400000, and the error will no longer occur.
Grigory
@pomadchin
Oh mb that was some runtime exception which has not been properly handled by Spark and exceptions are non serializable
Mr.Gordon
@GofferdoXu
Hey @pomadchin How to convert RDD to Tile? I know stitch at the moment.
Grigory
@pomadchin

@GofferdoXu depends on what you need

stitch stitches everything into a single raster on a driver => a pretty expensive operation

Mr.Gordon
@GofferdoXu
Is there any other way? @pomadchin
Grigory
@pomadchin
@GofferdoXu what way you need?
you can save them as chips without stitching
Mr.Gordon
@GofferdoXu
This is my code
val tile = value
      .where(Intersects(extent.center))
      .where(Between(ZonedDateTime.of(2017, 12, 27, 7, 0, 0, 0, ZoneOffset.UTC), ZonedDateTime.of(2021, 12, 28, 9, 0, 0, 0, ZoneOffset.UTC)))
      .result.sortByKey(false).map { case (key, tile) => (key.getComponent[SpatialKey], tile) }.reduceByKey(_.merge(_)).stitch()
    val png = tile.renderPng()
    png
@pomadchin I want to generate png in real time and display it on the map
Grigory
@pomadchin
so usually; you need to use Value reader in this case
and if you have the TMS map than each key would correspond to a tile on a map
Mr.Gordon
@GofferdoXu
But I want to perform a query operation😅
Grigory
@pomadchin
in this case that is the only way
if you’re sure that the result will always be small you can use the CollectionLayerReader for the temporal query to make it faster; the API would be the same but you wont need Spark
Mr.Gordon
@GofferdoXu
@pomadchin Thank you.
Grigory
@pomadchin
@GofferdoXu :+1:
Mr.Gordon
@GofferdoXu
Hey @pomadchin I still have a question. I use the pipeline, and the picture that comes out looks like this, and it doesn’t look normal.
image.png
😅
Grigory
@pomadchin
hey @GofferdoXu what do you mean by “doesnt look normal” ?
how should it look like?
Mr.Gordon
@GofferdoXu
@pomadchin I think it should be a remote sensing image, not such a dotted image
😅
Grigory
@pomadchin
not a ‘dotted' image?
looks pretty much normal to me
I guess to get some support from me you’d need to share 1. the source, 2. the set of operations you’re doing and 3. the result you’re gettings vs expecting
Mr.Gordon
@GofferdoXu
@pomadchin Thank you, let me study it first😅😅😅
wongjimsan
@wongjimsan_gitlab
Add ColorInterp support into GeoTiffOptions and GeoTiffWriter / GeoTiffReader #3020
Grigory
@pomadchin
hey @wongjimsan_gitlab ! that is not on our TODOs list for now; but I will help you in case you’d like to create a PR; and also - have you tried the color space param?
yang162132
@yang162132
Hey! @pomadchin. How to use the Branch in https://github.com/geotrellis/maml/blob/develop/shared/src/main/scala/ast/Expression.scala#L165. I think it hard to understand
Grigory
@pomadchin
hey @yang162132 the first expression in the list should be the condition (expression that returns boolean)
what example do you need btw? and example of a json or the case classes usage?
and what do you want to do there? mb I can help basing on your question
yang162132
@yang162132
@pomadchin Oh Thank you I want to an example of a json to know how it work such as If I want to do addition or multiplication by giving a condition could it achieve?
yang162132
@yang162132
And by the way is there a function to make two layers with a band to one layer with two bands
林海听枫
@stjimreal
Hello! How could I reproject a MultibandTileLayerRDD? When trying geotrellis-landsat-tutorial, I simply migrate to a newer "org.locationtech.geotrellis" %% "geotrellis-spark" % "3.5.2", however it may be incompatible to reproject a RasterRDD.
And how to read from a pyramid type datasat using "pipeline json string"?
Grigory
@pomadchin
hey @stjimreal this tutorial is just very outdated :/ probably imports are messed up
林海听枫
@stjimreal
@pomadchin I did make it to update the APIs to GT3.x. Well, I just connot fetch out the way to reproject a MultibandTileLayerRDD. Is it because the lack of some other packages?
林海听枫
@stjimreal
Here is my ingest code, it is for a multiband geotiff. It works, but how to reproject a MultibandTileLayerRDD?
def run(implicit sc: SparkContext): Unit = {
    val inputRdd: RDD[RasterSource] = {
    sc.parallelize(inputPath, inputPath.length)
      .map(uri => RasterSource(uri))
      .cache()
    }
    val layoutScheme = ZoomedLayoutScheme(WebMercator, tileSize = 256)
    val summary = RasterSummary.fromRDD(inputRdd)
    val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
    val tiled:MultibandTileLayerRDD[SpatialKey] = RasterSourceRDD
        .tiledLayerRDD(inputRdd, layout)
// We need to reproject the tiles to WebMercator
//    TileRDDReproject(tiled, WebMercator, layoutScheme, Some(Bilinear))
    val attributeStore = FileAttributeStore(outputPath)
    val writer = FileLayerWriter(attributeStore)
    Pyramid.fromLayerRDD(tiled, Some(zoom), Some(0)).write("landsat",
      writer,
      ZCurveKeyIndexMethod)
  }
Grigory
@pomadchin
Hey! Didn’t have a chance to look into your question but will do so shortly
林海听枫
@stjimreal

Hey! Didn’t have a chance to look into your question but will do so shortly

@pomadchin Thx! Would you like to conclude all the ways to ingest/output geotiffs and do ETL things? According to my exploring on the internet, there may have three ways like 1) RasterLayerRDD func 2) MAML 3) pipeline DSL. Where could I find a new DEMO with GT3.x on some web attributes?

Grigory
@pomadchin

hey @stjimreal;

we wanted to avoid terms ETL and to provide a good syntax so it is easy to build own pipelines.
RasterSources syntax and DSL is some sort of a nice replacement to the ETL.

However, Pipeline project is the closes to the old ETL one.

MAML is about lazy raster and rdds transformations

GT server uses GT 3.6 https://github.com/geotrellis/geotrellis-server/tree/main/stac-example

Jan Van den bosch
@bossie
good afternoon my friends
I'm instantiating Geotiff*RasterSources with an s3:// URI; that used to work fine but now I have to support reading geotiffs in buckets in different regions at the same time. How do I tell the underlying S3Client with which region it should communicate?
Grigory
@pomadchin
hey @bossie you can configure client via the S3ClientProducer.set method
Jan Van den bosch
@bossie
yes but that singleton is not going to work in this multithreaded application