geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Grigory

@pomadchin

:+1:

you’re welcome

Grigory

@pomadchin

@StrongSteve ohhh I tracked done the issue; it is indeed #3324 so only the current snapshot works

I’ll make a new release by the end of the week; for now you can use 3.5.2-SNAPSHOT; it lives in the eclipse snapshots repo:

resolvers ++= Seq(
  "eclipse-releases" at "https://repo.eclipse.org/content/groups/releases",
  "eclipse-snapshots" at "https://repo.eclipse.org/content/groups/snapshots"
)

StrongSteve

@StrongSteve

awesome! thx a lot!

StrongSteve

@StrongSteve

@pomadchin
confirmed -> snapshot fixes the issue! thx a lot!

Grigory

@pomadchin

@StrongSteve :+1: this one is a super weird bug (meaning - why it worked before?); it looks like all gt 3.x version are affected; however, it started to fail only starting some JVM version which is really not expected (by me)

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

hello. I try to read the tiff by GeoTiffReader.readSingleband or RasterSource api, but all of them went wrong.

it take

Exception in thread "main" java.lang.ExceptionInInitializerError
    at demo.Main$.main(Main.scala:73)
    at demo.Main.main(Main.scala)
Caused by: geotrellis.raster.io.geotiff.reader.MalformedGeoTiffException: incorrect byte order
    at geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.read(GeoTiffInfo.scala:132)
    at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readMultiband(GeoTiffReader.scala:213)
    at geotrellis.raster.geotiff.GeoTiffRasterSource$$anonfun$tiff$1.apply(GeoTiffRasterSource.scala:37)
    at geotrellis.raster.geotiff.GeoTiffRasterSource$$anonfun$tiff$1.apply(GeoTiffRasterSource.scala:37)
    at scala.Option.getOrElse(Option.scala:121)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff$lzycompute(GeoTiffRasterSource.scala:37)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff(GeoTiffRasterSource.scala:34)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent$lzycompute(GeoTiffRasterSource.scala:55)
    at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent(GeoTiffRasterSource.scala:55)
    at geotrellis.raster.RasterMetadata$class.extent(RasterMetadata.scala:52)
    at geotrellis.raster.RasterSource.extent(RasterSource.scala:44)
    at demo.package$.<init>(package.scala:16)
    at demo.package$.<clinit>(package.scala)

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

the library dependencies:

libraryDependencies ++= Seq(
  "org.locationtech.geotrellis" %% "geotrellis-raster" % "3.2.1-SNAPSHOT",
  "org.locationtech.geotrellis" %% "geotrellis-spark" % "3.2.1-SNAPSHOT",
  "org.apache.spark"      %% "spark-core"       % "2.2.0" % Provided,
  "org.scalatest"         %%  "scalatest"       % "2.2.0" % Test
)

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

When I tried to get the incorrent byte order by byteReader.get.toChar, I got the 'v' not 'I' or 'M'.

Grigory

@pomadchin

@DDDDDDDDDDDDLANG hm, it looks like you have TIFFs that are not supported by the GT Java reader, could you share it?

You can also try to use GDALRasterSource to workaround this issue

Max

@Max-AR

Hi! I have 2-3 geotiffs that overlap for a single tile. The three geotiffs are adjacent to one another in their extents (i.e not overlapping).

I am using the MosaicRasterSource to attempt to create the tile from the 2-3 geotiffs. My code looks like this:

    // Results in java.lang.ArrayIndexOutOfBoundsException: null 
    mosaic.reproject(targetCRS, strategy=Auto(0)).read(key.extent(layout))

I have also tried this code:

    mosaic.reproject(targetCRS, strategy=Auto(0)).tileToLayout(layout).read(key)

Which results in this error:

java.lang.IllegalArgumentException: requirement failed: chunk at GridBounds(0,0,255,17946) exceeds tile boundary at (256, 256)

I probably have some kind of fundamental misunderstanding on how to create these tiles. Any help would be appreciated! And is there anything in the documenation I can have a look at?

Max

@Max-AR

My data kind of looks like this, if it makes any difference:

+------------+
|XXXXXXYYYYYY|
|XXXXXXYYYYYY|
|XXXXXXYYYYYY|
|XXXXXXZZZZZZ|
|XXXXXXZZZZZZ|
|XXXXXXZZZZZZ|
+------------+

Grigory

@pomadchin

Hey @Max-AR how do you create the mosaic raster source?

Are all tiffs originally in the same projection?

Max

@Max-AR

    val rasterSources: Option[NonEmptyList[RasterSource]] = NonEmptyList.fromList(rs.toList)
    val mosaic = rasterSources match {
      case Some(x) =>
        val gridExtents = rs.head.gridExtent combine rs.toList(1).gridExtent
        MosaicRasterSource(x, requestScheme.crs, gridExtents)
      case None => throw new InternalError("Unable to create mosaic")
    }

In this case there are only two geotiffs, very WIP stuff haha

I was getting inspiration from here - https://github.com/locationtech/geotrellis/blob/master/raster/src/test/scala/geotrellis/raster/MosaicRasterSourceSpec.scala

Max

@Max-AR

Are all tiffs originally in the same projection?

Originally all the GeoTiffs are in EPSG:4326 and I want to convert to WebMercator

rs is passed into this function as Iterable[RasterSource]

Grigory

@pomadchin

@Max-AR try the following

 val mosaic = 
  MosaicRasterSource
    .instance(NonEmptyList.fromListUnsafe(rs.toList), WebMercator)

yea well the spec is a bit more specific about grids

However it may still fail; a while ago I captured the following bug: locationtech/geotrellis#3253

I'd say this one is not a trivial bug and we don't have any ETA for it to be fixed ):

Max

@Max-AR

Yeah, I am getting None returned from the .read(key). Is there another way I am able to emulate the same behaviour?

And thanks for helping me out :)

Grigory

@pomadchin

@Max-AR oh right so you are also getting Nons by reading keys that should be there?

eh I think that is the only way; otherwise yo can craft smth yourself

i.e. tileToLayout each raster source, read from it a tile by key and after that merge them together

there is smth wrong with how tileToLayout behaves with the MosaicRasterSource

for regular RasterSources it should work

tbh I would be very happy if we would address locationtech/geotrellis#3253 :D at some point

Max

@Max-AR

oh right so you are also getting Nons by reading keys that should be there?

Yeah, the three geotiffs are adjacent and when I create them individually in RasterSource it returns a partial tile with a black band when i use renderJpg or clear with renderPng. Hopefully that helps you debug the issue at all in the future :)

I'll give it a go reading the tiles individually and then merging them together over the areas with no data in them

I take it all the functionality to do what I need is here?

https://github.com/locationtech/geotrellis/tree/master/raster/src/main/scala/geotrellis/raster/merge

Max

@Max-AR

Thanks again for pointing me in the right direction :)

Grigory

@pomadchin

@Max-AR yep that's it, merge should work correct if you're mering tiles retrieved by the same key!

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG hm, it looks like you have TIFFs that are not supported by the GT Java reader, could you share it?

I have found some problems with my image data and I am trying to adjust it. Thank you very much. And the GDALRasterSource can also work.

Grigory

@pomadchin

@DDDDDDDDDDDDLANG you're welcome :+1:

tosen1990

@tosen1990

Anyone knows if there a direct way to mask a tile by certain values? I can only think out this:

    val array = Array(1,3,9,10)
    tif.mapTile(tile => {
      tile.mapIfSet(f => {
        if(array.contains(f)) NODATA
        else f
      })
    })

Grigory

@pomadchin

@tosen1990 I think given the current state of the API this is the best way of doing it.

Simeon H.K. Fitch

@metasim

📣📣 If you're a user of RasterFrames (the love-child of GeoTrellis & Spark SQL) and would like to see it kept up to date with the latest versions of Spark and JVM technologies, there's a small favor you could do us to help. Go to issue SPARK-7768 and vote for it.

The TL;DR of it is that RasterFrames and GeoMesa (and other frameworks built on Spark using UDTs) use a non-sustainable hack to register the types with Spark. This hack would no longer be required if the Spark committers changed literally one line of code. This ticket has been open since 2015 against Spark 1.5 and keeps getting pushed to the next release.

Regrettably, to vote for the issue you have to create an account on the Spark Jira system, but my hope is that collectively overcoming that small bit of friction will reap larger rewards.

2 replies

Grigory

@pomadchin

@metasim done! let me know if I can do smth else to help.

Simeon H.K. Fitch

@metasim

:pray:

Where communities thrive

People

Repo info

Activity