Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 22 19:41
    jisantuc review_requested #3330
  • Jan 22 19:41
    jisantuc synchronize #3330
  • Jan 22 17:48

    pomadchin on master

    Add rasters S3 write methods (#… (compare)

  • Jan 22 17:48
    pomadchin closed #3333
  • Jan 22 15:18
    echeipesh synchronize #3333
  • Jan 22 15:13

    echeipesh on master

    Cleanup GDALDataset, add initia… Cleanup the code Scala 2.11 compat (compare)

  • Jan 22 15:13
    echeipesh closed #3331
  • Jan 22 15:11
    echeipesh commented #3332
  • Jan 21 15:30
    echeipesh closed #3097
  • Jan 12 00:42
    pomadchin edited #3333
  • Jan 12 00:42
    pomadchin edited #3333
  • Jan 12 00:42
    pomadchin synchronize #3333
  • Jan 12 00:20
    pomadchin synchronize #3333
  • Jan 12 00:19
    pomadchin synchronize #3333
  • Jan 12 00:18
    pomadchin synchronize #3333
  • Jan 12 00:17
    pomadchin opened #3333
  • Jan 08 17:40
    pomadchin edited #3331
  • Jan 08 00:34
    pomadchin synchronize #3331
  • Jan 08 00:29
    pomadchin synchronize #3331
  • Jan 08 00:28
    pomadchin synchronize #3331
Grigory
@pomadchin
:+1:
you’re welcome
Grigory
@pomadchin
@StrongSteve ohhh I tracked done the issue; it is indeed #3324 so only the current snapshot works
I’ll make a new release by the end of the week; for now you can use 3.5.2-SNAPSHOT; it lives in the eclipse snapshots repo:
resolvers ++= Seq(
  "eclipse-releases" at "https://repo.eclipse.org/content/groups/releases",
  "eclipse-snapshots" at "https://repo.eclipse.org/content/groups/snapshots"
)
StrongSteve
@StrongSteve
awesome! thx a lot!
StrongSteve
@StrongSteve
image.png
@pomadchin
confirmed -> snapshot fixes the issue! thx a lot!
Grigory
@pomadchin
@StrongSteve :+1: this one is a super weird bug (meaning - why it worked before?); it looks like all gt 3.x version are affected; however, it started to fail only starting some JVM version which is really not expected (by me)
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG
hello. I try to read the tiff by GeoTiffReader.readSingleband or RasterSource api, but all of them went wrong.
it take
Exception in thread "main" java.lang.ExceptionInInitializerError
    at demo.Main$.main(Main.scala:73)
    at demo.Main.main(Main.scala)
Caused by: geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: incorrect byte order
    at geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.read(GeoTiffInfo.scala:132)
    at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readMultiband(GeoTiffReader.scala:213)
    at geotrellis.raster.geotiff.GeoTiffRasterSource$$anonfun$tiff$1.apply(GeoTiffRasterSource.scala:37)
    at geotrellis.raster.geotiff.GeoTiffRasterSource$$anonfun$tiff$1.apply(GeoTiffRasterSource.scala:37)
    at scala.Option.getOrElse(Option.scala:121)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff$lzycompute(GeoTiffRasterSource.scala:37)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff(GeoTiffRasterSource.scala:34)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent$lzycompute(GeoTiffRasterSource.scala:55)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent(GeoTiffRasterSource.scala:55)
    at geotrellis.raster.RasterMetadata$class.extent(RasterMetadata.scala:52)
    at geotrellis.raster.RasterSource.extent(RasterSource.scala:44)
    at demo.package$.<init>(package.scala:16)
    at demo.package$.<clinit>(package.scala)
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG
the library dependencies:
libraryDependencies ++= Seq(
  "org.locationtech.geotrellis" %% "geotrellis-raster" % "3.2.1-SNAPSHOT",
  "org.locationtech.geotrellis" %% "geotrellis-spark" % "3.2.1-SNAPSHOT",
  "org.apache.spark"      %% "spark-core"       % "2.2.0" % Provided,
  "org.scalatest"         %%  "scalatest"       % "2.2.0" % Test
)
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG
When I tried to get the incorrent byte order by byteReader.get.toChar, I got the 'v' not 'I' or 'M'.
Grigory
@pomadchin
@DDDDDDDDDDDDLANG hm, it looks like you have TIFFs that are not supported by the GT Java reader, could you share it?
You can also try to use GDALRasterSource to workaround this issue
Max
@Max-AR

Hi! I have 2-3 geotiffs that overlap for a single tile. The three geotiffs are adjacent to one another in their extents (i.e not overlapping).

I am using the MosaicRasterSource to attempt to create the tile from the 2-3 geotiffs. My code looks like this:

    // Results in java.lang.ArrayIndexOutOfBoundsException: null 
    mosaic.reproject(targetCRS, strategy=Auto(0)).read(key.extent(layout))

I have also tried this code:

    mosaic.reproject(targetCRS, strategy=Auto(0)).tileToLayout(layout).read(key)

Which results in this error:

java.lang.IllegalArgumentException: requirement failed: chunk at GridBounds(0,0,255,17946) exceeds tile boundary at (256, 256)

I probably have some kind of fundamental misunderstanding on how to create these tiles. Any help would be appreciated! And is there anything in the documenation I can have a look at?
Max
@Max-AR
My data kind of looks like this, if it makes any difference:
+------------+
|XXXXXXYYYYYY|
|XXXXXXYYYYYY|
|XXXXXXYYYYYY|
|XXXXXXZZZZZZ|
|XXXXXXZZZZZZ|
|XXXXXXZZZZZZ|
+------------+
Grigory
@pomadchin
Hey @Max-AR how do you create the mosaic raster source?
Are all tiffs originally in the same projection?
Max
@Max-AR
    val rasterSources: Option[NonEmptyList[RasterSource]] = NonEmptyList.fromList(rs.toList)
    val mosaic = rasterSources match {
      case Some(x) =>
        val gridExtents = rs.head.gridExtent combine rs.toList(1).gridExtent
        MosaicRasterSource(x, requestScheme.crs, gridExtents)
      case None => throw new InternalError("Unable to create mosaic")
    }
In this case there are only two geotiffs, very WIP stuff haha
Max
@Max-AR

Are all tiffs originally in the same projection?

Originally all the GeoTiffs are in EPSG:4326 and I want to convert to WebMercator

rs is passed into this function as Iterable[RasterSource]
Grigory
@pomadchin
@Max-AR try the following
 val mosaic = 
  MosaicRasterSource
    .instance(NonEmptyList.fromListUnsafe(rs.toList), WebMercator)
yea well the spec is a bit more specific about grids
However it may still fail; a while ago I captured the following bug: locationtech/geotrellis#3253
I'd say this one is not a trivial bug and we don't have any ETA for it to be fixed ):
Max
@Max-AR
Yeah, I am getting None returned from the .read(key). Is there another way I am able to emulate the same behaviour?
And thanks for helping me out :)
Grigory
@pomadchin
@Max-AR oh right so you are also getting Nons by reading keys that should be there?
eh I think that is the only way; otherwise yo can craft smth yourself
i.e. tileToLayout each raster source, read from it a tile by key and after that merge them together
there is smth wrong with how tileToLayout behaves with the MosaicRasterSource
for regular RasterSources it should work
tbh I would be very happy if we would address locationtech/geotrellis#3253 :D at some point
Max
@Max-AR

oh right so you are also getting Nons by reading keys that should be there?

Yeah, the three geotiffs are adjacent and when I create them individually in RasterSource it returns a partial tile with a black band when i use renderJpg or clear with renderPng. Hopefully that helps you debug the issue at all in the future :)

I'll give it a go reading the tiles individually and then merging them together over the areas with no data in them
I take it all the functionality to do what I need is here?
Max
@Max-AR
Thanks again for pointing me in the right direction :)
Grigory
@pomadchin
@Max-AR yep that's it, merge should work correct if you're mering tiles retrieved by the same key!
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG hm, it looks like you have TIFFs that are not supported by the GT Java reader, could you share it?

I have found some problems with my image data and I am trying to adjust it. Thank you very much. And the GDALRasterSource can also work.

Grigory
@pomadchin
@DDDDDDDDDDDDLANG you're welcome :+1:
tosen1990
@tosen1990
Anyone knows if there a direct way to mask a tile by certain values? I can only think out this:
    val array = Array(1,3,9,10)
    tif.mapTile(tile => {
      tile.mapIfSet(f => {
        if(array.contains(f)) NODATA
        else f
      })
    })
Grigory
@pomadchin
@tosen1990 I think given the current state of the API this is the best way of doing it.
Simeon H.K. Fitch
@metasim

📣📣 If you're a user of RasterFrames (the love-child of GeoTrellis & Spark SQL) and would like to see it kept up to date with the latest versions of Spark and JVM technologies, there's a small favor you could do us to help. Go to issue SPARK-7768 and vote for it.

The TL;DR of it is that RasterFrames and GeoMesa (and other frameworks built on Spark using UDTs) use a non-sustainable hack to register the types with Spark. This hack would no longer be required if the Spark committers changed literally one line of code. This ticket has been open since 2015 against Spark 1.5 and keeps getting pushed to the next release.

Regrettably, to vote for the issue you have to create an account on the Spark Jira system, but my hope is that collectively overcoming that small bit of friction will reap larger rewards.

2 replies
Grigory
@pomadchin
@metasim done! let me know if I can do smth else to help.
Simeon H.K. Fitch
@metasim
:pray: