geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Mr.Gordon

@GofferdoXu

def createSinglebandTileJson(input: String, output: String, layerName: String): Unit = {
    val maskJson: String =
      """
        |[
        |  {
        |    "uri" : "{input}",
        |    "time_tag" : "TIFFTAG_DATETIME",
        |    "time_format" : "yyyy:MM:dd HH:mm:ss",
        |    "type" : "singleband.spatial.read.hadoop"
        |  },
        |  {
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.tile-to-layout"
        |  },
        |  {
        |    "crs" : "EPSG:3857",
        |    "scheme" : {
        |      "crs" : "epsg:3857",
        |      "tileSize" : 256,
        |      "resolutionThreshold" : 0.1
        |    },
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.buffered-reproject"
        |  },
        |  {
        |    "end_zoom" : 0,
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.pyramid"
        |  },
        |  {
        |    "name" : "{layerName}",
        |    "uri" : "{output}",
        |    "key_index_method" : {
        |      "type" : "zorder",
        |      "temporal_resolution": 1
        |    },
        |    "scheme" : {
        |      "crs" : "epsg:3857",
        |      "tileSize" : 256,
        |      "resolutionThreshold" : 0.1
        |    },
        |    "type" : "singleband.temporal.write"
        |  }
        |]
      """.stripMargin
    val maskJsonStr = maskJson.replace("{input}", input).replace("{output}", output).replace("{layerName}", layerName)
    val list: Option[Node[Stream[(Int, TileLayerRDD[SpaceTimeKey])]]] = maskJsonStr.node
    list match {
      case None => println("Couldn't parse the JSON")
      case Some(node) => {
        node.eval.foreach { case (zoom, rdd) =>
          println(s"ZOOM: ${zoom}")
          println(s"COUNT: ${rdd.count}")
        }
      }
    }

Mr.Gordon

@GofferdoXu

def createSinglebandTileJson(input: String, output: String, layerName: String): Unit = {
    val maskJson: String =
      """
        |[
        |  {
        |    "uri" : "{input}",
        |    "time_tag" : "TIFFTAG_DATETIME",
        |    "time_format" : "yyyy:MM:dd HH:mm:ss",
        |    "type" : "singleband.spatial.read.hadoop"
        |  },
        |  {
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.tile-to-layout"
        |  },
        |  {
        |    "crs" : "EPSG:3857",
        |    "scheme" : {
        |      "crs" : "epsg:3857",
        |      "tileSize" : 256,
        |      "resolutionThreshold" : 0.1
        |    },
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.buffered-reproject"
        |  },
        |  {
        |    "end_zoom" : 0,
        |    "resample_method" : "nearest-neighbor",
        |    "type" : "singleband.spatial.transform.pyramid"
        |  },
        |  {
        |    "name" : "{layerName}",
        |    "uri" : "{output}",
        |    "key_index_method" : {
        |      "type" : "zorder",
        |      "temporal_resolution": 1
        |    },
        |    "scheme" : {
        |      "crs" : "epsg:3857",
        |      "tileSize" : 256,
        |      "resolutionThreshold" : 0.1
        |    },
        |    "type" : "singleband.temporal.write"
        |  }
        |]
      """.stripMargin
    val maskJsonStr = maskJson.replace("{input}", input).replace("{output}", output).replace("{layerName}", layerName)
    val list: Option[Node[Stream[(Int, TileLayerRDD[SpaceTimeKey])]]] = maskJsonStr.node
    list match {
      case None => println("Couldn't parse the JSON")
      case Some(node) => {
        node.eval.foreach { case (zoom, rdd) =>
          println(s"ZOOM: ${zoom}")
          println(s"COUNT: ${rdd.count}")
        }
      }
    }
  }

How to implement time series？

Grigory

@pomadchin

hey @GofferdoXu what time series do you talk about?

also on the read stage you have a typo: singleband.temporal.read.hadoop

just change all spatial prefixes to temporal

Mr.Gordon

@GofferdoXu

thank you

Mr.Gordon

@GofferdoXu

I wonder how to update layers in geotrellis3

Mr.Gordon

@GofferdoXu

hey@pomadchin How to implement layer update？When there are new images

Grigory

@pomadchin

@GofferdoXu I don’t think it is implemented in terms of the Pipeline DSL

but you can implement it yourself via GT API we have LayerUpdater

Eetu Pursiainen

@epursiai

Hello! I am currently trying to get GeoTrellis to work in Databricks, but I am facing some issue(s). I have installed geotrellis-spark and geotrellis-raster (versions 3.6.0) from the Databrick's UI through Maven. I have the following code:
import geotrellis.raster._
import geotrellis.spark._
import geotrellis.raster.io.geotiff._

val rasterFilePath = "path/to/my.tif"
val geoTiff = GeoTiffReader.readSingleband(rasterFilePath)

But I'm getting java.lang.NoClassDefFoundError: Could not initialize class geotrellis.raster.io.geotiff.TiffType$ error. I noticed that someone else had this problem here before, but unfortunately the solution was never posted here. Any clues what I'm doing wrong here?

Grigory

@pomadchin

hey @epursiai I think it can be the circe library version mismatch

try to shade it in your artifacts

Grigory

@pomadchin

We can also roll a GT release with the fresh circe version

do you know which version is needed for the Databricks runtime?

Eetu Pursiainen

@epursiai

I'll try to exclude circe from the dependencies and use what's provided in the runtime, and I'll let you know if it works. Unfortunately I have no clue what the version there is. I can try to find out. :)

Grigory

@pomadchin

@epursiai yes please! that would be very much welcome

I think shading can be a better approach

Eetu Pursiainen

@epursiai

Probably will need to do that for circe-core, circe-generic and circe-parser all?

Grigory

@pomadchin

try to shade the entire io.circe package

Eetu Pursiainen

@epursiai

Yeah, I'll try to do that. I'll update this issue tomorrow.

Grigory

@pomadchin

:+1: thank you

santocp94

@santocp94

hey @pomadchin I'm sorry but I'm still having some trouble with my mosaic task :( . I wrote this starting from your suggestion:

  val firstTiff = GeoTiffReader
    .readSingleband("...")
  val combinedExtent = firstExtent.expandToInclude(secondExtent)
  val re = RasterExtent(combinedExtent, firstTiff.cellSize)

  val newRaster = Raster(ArrayTile.alloc(firstTiff.cellType, re.cols, re.rows), re.extent)
  val mergedRaster = newRaster.merge(Raster(firstRaster.tile, firstExtent))

I can't understand why but the merge function with that configuration is not resolved. I tried with other two rasters read form files and the function is correctly resolved. The only difference I noticed between the created raster and the read ones is that the geotrellis.raster.DoubleUserDefinedNoDataArrayTile produces a merge while the read raster have a inner geotrellis.raster.io.geotiff.Float64GeoTiffTile. Did I mess up something with imports and packages?
Thanks in advance

Grigory

@pomadchin

hey @santocp94 what to you mean by not correctly resolved?

caoguangshun

@caoguangshun

ok @pomadchin ,i have sovled the problem referring to your test code in the github. and i add a configuration file in resource folder .as following

https://github.com/locationtech/geotrellis/blob/master/s3-spark/src/test/scala/geotrellis/spark/store/s3/MockS3Client.scala

Grigory

@pomadchin

@caoguangshun great!

santocp94

@santocp94

hey @santocp94 what to you mean by not correctly resolved?

I tried with different configurations but an error occurred:

value merge is not a member of geotrellis.raster.Raster[geotrellis.raster.MutableArrayTile]

value merge is not a member of geotrellis.raster.Raster[geotrellis.raster.ArrayTile]

Grigory

@pomadchin

@santocp94 try to cast internals of the Raster to MultibandTile

if that would not work, check if withMultibandRasterMergeMethod(raster).merge works

but I bet this is due to types mismatch

MutableArrayTile is kind of a dangerous type (:

I’m surprised that it didn’t work with ArrayTile

looks like some imports hell.

santocp94

@santocp94

Looks like I can't just cast it:

Exception in thread "main" java.lang.ClassCastException: class geotrellis.raster.DoubleUserDefinedNoDataArrayTile cannot be cast to class geotrellis.raster.MultibandTile

and I'm actually failing to import withMultibandRasterMergeMethod. I also tried to re-import everything from scratch but the problem persists

Grigory

@pomadchin

@santocp94 well yes, DoubleUserDefinedNoDataArrayTile can be cast to Tile

it is not a MultibandTile (:

@santocp94 what are your imports?

import geotrellis.raster._ should be able to get it into the scope

If that doesn’t happen - than or implicits are wrong or smth else is happening in your codebase

The last resort could be to use it directly from geotrellis.raster.merge.Implicits._ but I can recommend it only for test / dev purposes

Eetu Pursiainen

@epursiai

Hmm. So excluding circe from dependencies when installing Geotrellis in Databricks didn't work, as then those needed circe classes were not found at all. Since I'm not really a experienced Java/Scala dev, I'm wondering what's the easiest way to create a proper uber jar, or to shade those circe dependencies...

Grigory

@pomadchin

@epursiai just create an uber jar and shade them; check out https://github.com/sbt/sbt-assembly#shading

we usually use sbt-assembly plugin for these purposes; The example of usage in the gt build code is here https://github.com/locationtech/geotrellis/blob/master/project/Settings.scala#L559-L569

So in your case along with the circe rule you would need shapeless and cats kernel shading:

ShadeRule.rename("io.circe.**" -> s”$shadePackage.io.circe.@1").inAll
ShadeRule.rename("shapeless.**" -> s"$shadePackage.shapeless.@1").inAll,
ShadeRule.rename("cats.kernel.**" -> s"$shadePackage.cats.kernel.@1").inAll

yang162132

@yang162132

'./scripts/server --ndvi
Starting geotrellis-server-450_ndvi-example_1 ... done
Attaching to geotrellis-server-450_ndvi-example_1
ndvi-example_1 | Error: Could not find or load main class geotrellis.server.example.ndvi.NdviServer
geotrellis-server-450_ndvi-example_1 exited with code 1
' what's wrong

Grigory

@pomadchin

hey @yang162132 what are you trying to do?

what repo it is and what’s happening there

yang162132

@yang162132

I want to run the example in https://github.com/geotrellis/geotrellis-server

Grigory

@pomadchin

oooooh I see; I think I haven’t properly updated the docker compose file

for now I think it would be more stragihtforward to:

$ sbt
$ project example
$ // run

Where communities thrive

People

Repo info

Activity