Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 06:22
    dependabot[bot] labeled #3490
  • 06:22

    dependabot[bot] on npm_and_yarn

    Bump decode-uri-component from … (compare)

  • 06:22
    dependabot[bot] opened #3490
  • Dec 02 02:54
    pomadchin commented #3489
  • Dec 02 02:53
    pomadchin commented #3489
  • Dec 02 02:53
    pomadchin commented #3489
  • Dec 02 02:24
    imperio-wxm commented #3489
  • Nov 30 15:14
    pomadchin commented #3489
  • Nov 30 15:11
    pomadchin commented #3489
  • Nov 30 15:11
    pomadchin commented #3489
  • Nov 30 15:11
    pomadchin commented #3489
  • Nov 30 15:10
    pomadchin commented #3489
  • Nov 30 15:10
    pomadchin labeled #3489
  • Nov 30 15:10
    pomadchin labeled #3489
  • Nov 30 15:09
    pomadchin commented #3489
  • Nov 30 12:01
    imperio-wxm edited #3489
  • Nov 30 11:55
    imperio-wxm opened #3489
  • Nov 26 17:37
    pomadchin synchronize #3487
  • Nov 26 17:37
    pomadchin synchronize #3487
  • Nov 21 09:10
    jdries opened #3488
zmx110
@zmx110
@pomadchin I'm sorry to bother you, but I have a few questions chat privately with you. Can You Get Back to me?Thank you.
Grigory
@pomadchin
@goodjobtry what is the code you used to ingest data into HBase? that's quite interesting; defionitely some metadata mismatch is happening (: for sure a reproducible example is the best we can start with

also you know that it is possible to insert a scala code in here? it is a bit incoinvent to copy coode from screenshots;

there is an md syntax supporot:

```scala
code
```

goodjobtry
@goodjobtry
the code to ingest data in to Hbase in other Object
import geotrellis.proj4.LatLng
import geotrellis.raster.MultibandTile
import geotrellis.raster.resample.Bilinear
import geotrellis.spark.io.{SpatialKeyFormat, tileLayerMetadataFormat}
import geotrellis.spark.{LayerId, Metadata, MultibandTileLayerRDD, SpatialKey, TileLayerMetadata, withTilerMethods}
import geotrellis.spark.io.hadoop.HadoopSparkContextMethodsWrapper
import geotrellis.spark.io.hbase.{HBaseAttributeStore, HBaseInstance, HBaseLayerManager, HBaseLayerWriter}
import geotrellis.spark.io.index.ZCurveKeyIndexMethod
import geotrellis.spark.pyramid.Pyramid
import geotrellis.spark.tiling.{FloatingLayoutScheme, ZoomedLayoutScheme}
import geotrellis.vector.ProjectedExtent
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

import scala.io.StdIn

object IngestTiffToHbase{
  val inputPath = "hdfs://hadoop103:8020/tiff/LC08_L1TP_125035_20131204_20170428_01_T1_B7.TIF"
  def main(args: Array[String]): Unit = {
    // Setup Spark to use Kryo serializer.
    val conf =
      new SparkConf()
        .setMaster("local[*]")
        .setAppName("Spark Tiler")
        .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
        .set("spark.kryo.registrator", "geotrellis.spark.io.kryo.KryoRegistrator")

    val sc = new SparkContext(conf)
    try {
      run(sc)
      // Pause to wait to close the spark context,
      // so that you can check out the UI at http://localhost:4040
      println("Hit enter to exit.")
      StdIn.readLine()
    } finally {
      sc.stop()
    }
  }

  def fullPath(path: String) = new java.io.File(path).getAbsolutePath

  def run(implicit sc: SparkContext) = {
    // Read the geotiff in as a single image RDD,
    // using a method implicitly added to SparkContext by
    // an implicit class available via the
    // "import geotrellis.spark.io.hadoop._ " statement.
    val inputRdd: RDD[(ProjectedExtent, MultibandTile)] = {
      sc.hadoopMultibandGeoTiffRDD(inputPath)
    }

    // Use the "TileLayerMetadata.fromRdd" call to find the zoom
    // level that the closest match to the resolution of our source image,
    // and derive information such as the full bounding box and data type.
    val (_, rasterMetaData) =
    TileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))

    // Use the Tiler to cut our tiles into tiles that are index to a floating layout scheme.
    // We'll repartition it so that there are more partitions to work with, since spark
    // likes to work with more, smaller partitions (to a point) over few and large partitions.
    val tiled: RDD[(SpatialKey, MultibandTile)] =
    inputRdd
      .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
      .repartition(100)

    // We'll be tiling the images using a zoomed layout scheme
    // in the web mercator format (which fits the slippy map tile specification).
    // We'll be creating 256 x 256 tiles.
    val layoutScheme = ZoomedLayoutScheme(LatLng, tileSize = 256)

    // We need to reproject the tiles to WebMercator
    val (zoom, reprojected): (Int, RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]]) =
      MultibandTileLayerRDD(tiled, rasterMetaData)
        .reproject(LatLng, layoutScheme, Bilinear)

    // Create the attributes store that will tell us information about our catalog.
    var zookpeers: Seq[String] = List("hadoop102:2181")
    var master: String = "hadoop102"

    val hbaseInstance = HBaseInstance(zookpeers, master)
    val attributeStore = HBaseAttributeStore(hbaseInstance)

    // Create the writer that we will use to store the tiles in the local catalog.
    var catalog: String = "catalogLatLng"
    val writer = HBaseLayerWriter(attributeStore, catalog)

    // Pyramiding up the zoom levels, write our tiles out to the local file system.
    Pyramid.upLevels(reprojected, layoutScheme, zoom, Bilinear) { (rdd, z) =>
      val layerId = LayerId("landsat", z)
      // If the layer exists already, delete it out before writing
      if (attributeStore.layerExists(layerId)) {
        new HBaseLayerManag
In this object , Trying to ingest data into Hbase
goodjobtry
@goodjobtry
Other scala Object ,tring to get tile in Hbase ,the code as follows:
package service
import geotrellis.raster._
import geotrellis.spark._
import geotrellis.spark.io._
import geotrellis.spark.io.file._
import akka.actor._
import akka.event.{Logging, LoggingAdapter}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import akka.stream.{ActorMaterializer, Materializer}
import org.apache.hadoop.fs.Path

import scala.concurrent._
import com.typesafe.config.ConfigFactory
import geotrellis.raster.render.{ColorMap, ColorRamp, Png, RGB}
import geotrellis.spark.io.hadoop.{HadoopAttributeStore, HadoopLayerReader, HadoopValueReader, SerializableConfiguration}
import geotrellis.spark.io.hbase.{HBaseAttributeStore, HBaseInstance, HBaseLayerReader, HBaseValueReader}
import org.apache.hadoop.hbase.client.Admin
import org.apache.spark.{SparkConf, SparkContext}
import service.Serve.root

import java.net.URI

object ServeFromHbase extends App with Service1 {
  val conf =
    new SparkConf()
      .setMaster("local[*]")
      .setAppName("Spark Tiler")
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.kryo.registrator", "geotrellis.spark.io.kryo.KryoRegistrator")

  var zookpeers: Seq[String] = List("hadoop102:2181")
  var master: String = "hadoop102"
  val hbaseInstance = HBaseInstance(zookpeers, master)
  val attributeStore = HBaseAttributeStore.apply(hbaseInstance,"catalogLatLng")


  val colorRamp =
    ColorRamp(RGB(0,0,0), RGB(255,255,255))
      .stops(100)
      .setAlphaGradient(0xFF, 0xAA)

  override implicit val system = ActorSystem("tutorial-system")
  override implicit val executor = system.dispatcher
  override implicit val materializer = ActorMaterializer()
  override val logger = Logging(system, getClass)
  Http().bindAndHandle(root, "0.0.0.0", 8081)
}

trait Service1 {
  implicit val system: ActorSystem
  implicit def executor: ExecutionContextExecutor
  implicit val materializer: Materializer
  val logger: LoggingAdapter

  def pngAsHttpResponse(png: Png): HttpResponse =
    HttpResponse(entity = HttpEntity(ContentType(MediaTypes.`image/png`), png.bytes))

  def root =
  //    http://localhost:8080/6/16/25
    pathPrefix(IntNumber / IntNumber / IntNumber) { (zoom, x, y) =>
      complete {
        Future {
          implicit val sc = new SparkContext(ServeFromHbase.conf)
          // Read in the tile at the given z/x/y coordinates.
          val tileOpt: Option[Tile] =
            try {
              val key = SpatialKey(x,y)
              val layerId:LayerId = LayerId("landsat",zoom)
              val reader2 = HBaseValueReader.apply(ServeFromHbase.attributeStore)
              val tile: Tile = reader2.reader[SpatialKey, Tile](layerId: LayerId).read(key)
              //val reader3 = HBaseLayerReader(ServeFromHbase.attributeStore)
             // reader3.read(layerId,)
              Some(tile)
            } catch {
              case _: ValueNotFoundError =>
                None
            }finally{
              sc.stop()
            }
          tileOpt.map { tile =>
            // Render as a PNG
            val png = tile.renderPng()
            pngAsHttpResponse(png)
          }
        }
      }
    } ~
      pathEndOrSingleSlash {
        getFromFile("static/index.html")
      } ~
      pathPrefix("") {
        getFromDirectory("static")
      }
}
I don't know what the problem is
Grigory
@pomadchin
@goodjobtry indeed kinda weird; can you list layers in the attribute store?
goodjobtry
@goodjobtry
image.png
is this?
Grigory
@pomadchin
@goodjobtry yea, but I mean via the AttributeStore
Oh I see the issue, on write data is written into the metadata table; but on the server side catalogLatLng is pointed as a metadata table
val attributeStore = HBaseAttributeStore.apply(hbaseInstance,"catalogLatLng") => replace to val attributeStore = HBaseAttributeStore.apply(hbaseInstance)
goodjobtry
@goodjobtry
I'll give it a try
How do I know to read which table in Hbase?
goodjobtry
@goodjobtry
image.png
Where should I specify the table name
goodjobtry
@goodjobtry
@pomadchin
Grigory
@pomadchin
@goodjobtry you don’t need that; the idea is to create a value reader with an attribute store that points to the hbase metadata table and that’s it
by inspecting the metadata table contents it’s possible to notice that it contins info about the tile table location
goodjobtry
@goodjobtry
but when i relplaced ,there also have a error
image.png
@pomadchin
Grigory
@pomadchin
@goodjobtry that’s a bit truncated error (: I’d tell smth shrinks the stacktrace, could you get a better one? But I doubt that is related to GeoTrellis
goodjobtry
@goodjobtry
It means that I can not get tile in Hbase and published as a TMS service?
@pomadchin
Grigory
@pomadchin
yes, some error happened and soome npe exception has been thrown
goodjobtry
@goodjobtry
The corresponding tile data must be found through the attribute table, rather than reading the tile directly from the table where the tile is stored
4 replies
Is that so
@pomadchin
tosen1990
@tosen1990

@pomadchin Hi, Grigory. Do you know if there is a workaround to get the same read performance in RasterFrames ?

From my test, GT raster reading using partitionBytes is several times faster than RF using .withSpatialIndex(partitionnum) ?

    val options =
      HadoopGeoTiffRDD.Options(
        numPartitions = Option(1000),
        partitionBytes = Option(64l * 1024 * 1024)
      )
    val geoTiffRDD: RDD[(ProjectedExtent, MultibandTile)] = HadoopGeoTiffRDD
      .spatialMultiband(new Path(inputPath), options)(spark.sparkContext)
    val df_init: DataFrame = spark.read.raster
      .withSpatialIndex(partitionnum)
      .withTileDimensions(256, 256)
      .withBandIndexes(bandIndexInt: _*)
      .fromCSV(cat).load()
      .withColumn("tile_dimensions", rf_dimensions(col(firstBandTile)))
3 replies
goodjobtry
@goodjobtry
@pomadchin Here's a new question about get tile from Hbase , I can't resolve it,Can you give me a suggestion?
org.apache.avro.AvroTypeException: Found geotrellis.raster.ArrayMultibandTile, expecting union. This can be caused by using a type parameter which doesn't match the object being deserialized.
I didn't change the code, I just reran the code
4 replies
yang162132
@yang162132
@pomadchin Hey! how could I get a tiff in fixed pixel scale. I had try
Raster(MbTile.resample(rasterExtent, Math.round(rasterExtent.width / 30).toInt, Math.round(rasterExtent.height / 30).toInt, NearestNeighbor), rasterExtent)
to get a 30m/pixel tiff
But finally I just get 26m/pixel
2 replies
yang162132
@yang162132
And it seems that the number of pixels has increased by about 30% from zoom =12(about 38.21meter) to 30 meter
mwhei
@mwhei
Hello, I store data into Hbase via " val layerId = LayerId("testid", z) writer.write(layerId, rdd, keyIndex)".But when fetch data via
" reader.readSpaceTimeKey, Tile, TileLayerMetadata[SpaceTimeKey])" .It has error. Exception in thread "main" geotrellis.store.package$LayerNotFoundError: Layer Layer(name = "testid", zoom = 0) not found in the catalog
2 replies
Is there some whole examples about how to store and fetch tif to/from hbase?
yang162132
@yang162132
Hey!@pomadchin
I've been thinking about a question recently.
which is the best choice for "tiff -> Map_algebra ->tiff" in GT for some big tif?
when I try readMultiband to operate directly on tiff it's really slow(it took about 15 minutes for reproject without other calculate).
and if create gt cataloge in advance,it will quickly in calculat.But it hard to load the result from the memory to a geotiff, both stitch from TileLayerRDD and meger to collect all tiles for GeoTrellisRasterSource expensive(both them are unsupport multi thread I guess).They also faced difficulty to get the pixel scale we want because the layer in zoom has fixed resolution.
So how can we make the most of our machines.
tosen1990
@tosen1990
Using crop on RDD to output the image, I got two different results.
Output-1 using stitch works well. While using foreach to write out, the image got distorted.
   val floatingLayout = FloatingLayoutScheme(512, 512)
    val tlm: TileLayerMetadata[SpatialKey] = inputRDD.collectMetadata[SpatialKey](floatingLayout)._2

    val tiled: RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]] = inputRDD.tileToLayout(tlm)

    val cropRDD: RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]] = {
      tiled.crop(targetExtent)
    }

    // it's okay       output-1
    val stitchedTile: Raster[MultibandTile] = cropRDD.crop(targetExtent).stitch()

    val croppedTile = stitchedTile  //.crop(totalCols, totalRows)
    MultibandGeoTiff(croppedTile.tile, croppedTile.extent,
      tlm.crs, GeoTiffOptions.DEFAULT.copy(compression = DeflateCompression))
      .withOverviews(NearestNeighbor)
      .write(new Path("xxx"), hconf.value)

    // the image distorted  output-2
    cropRDD.regrid(512, 512).toGeoTiffs()
      .filter{
        case (_: SpatialKey, null) ⇒ false  // remove any null Tiles
        case _ ⇒ true
      }
      .foreach { case (sk: SpatialKey, gt: MultibandGeoTiff) ⇒
        val path = new Path(new Path("xxx"), s"${sk.col}_${sk.row}.tif")
        MultibandGeoTiff(gt.tile, gt.extent,
          tlm.crs, GeoTiffOptions.DEFAULT.copy(compression = DeflateCompression))
          .withOverviews(NearestNeighbor)
          .write(path, hconf.value)
      }
output-1
WeChat1d8930ec7b50645eb156f78e8eb229e1.png
output2
WeChat9e76e8d97456c8c79eb6d3caedd435ff.png