GeoTrellis is a geographic data processing engine for high performance applications.
@pomadchin Hi, Grigory. Do you know if there is a workaround to get the same read performance in RasterFrames ?
From my test, GT raster reading using partitionBytes is several times faster than RF using .withSpatialIndex(partitionnum) ?
val options =
HadoopGeoTiffRDD.Options(
numPartitions = Option(1000),
partitionBytes = Option(64l * 1024 * 1024)
)
val geoTiffRDD: RDD[(ProjectedExtent, MultibandTile)] = HadoopGeoTiffRDD
.spatialMultiband(new Path(inputPath), options)(spark.sparkContext) val df_init: DataFrame = spark.read.raster
.withSpatialIndex(partitionnum)
.withTileDimensions(256, 256)
.withBandIndexes(bandIndexInt: _*)
.fromCSV(cat).load()
.withColumn("tile_dimensions", rf_dimensions(col(firstBandTile)))
3 replies
4 replies
Raster(MbTile.resample(rasterExtent, Math.round(rasterExtent.width / 30).toInt, Math.round(rasterExtent.height / 30).toInt, NearestNeighbor), rasterExtent)
2 replies
I've been thinking about a question recently.
which is the best choice for "tiff -> Map_algebra ->tiff" in GT for some big tif？
when I try readMultiband to operate directly on tiff it's really slow(it took about 15 minutes for reproject without other calculate).
and if create gt cataloge in advance,it will quickly in calculat.But it hard to load the result from the memory to a geotiff, both stitch from TileLayerRDD and meger to collect all tiles for GeoTrellisRasterSource expensive(both them are unsupport multi thread I guess).They also faced difficulty to get the pixel scale we want because the layer in zoom has fixed resolution.
So how can we make the most of our machines.
1 reply
Output-1 using
stitch works well. While using foreach to write out, the image got distorted. val floatingLayout = FloatingLayoutScheme(512, 512)
val tlm: TileLayerMetadata[SpatialKey] = inputRDD.collectMetadata[SpatialKey](floatingLayout)._2
val tiled: RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]] = inputRDD.tileToLayout(tlm)
val cropRDD: RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]] = {
tiled.crop(targetExtent)
}
// it's okay output-1
val stitchedTile: Raster[MultibandTile] = cropRDD.crop(targetExtent).stitch()
val croppedTile = stitchedTile //.crop(totalCols, totalRows)
MultibandGeoTiff(croppedTile.tile, croppedTile.extent,
tlm.crs, GeoTiffOptions.DEFAULT.copy(compression = DeflateCompression))
.withOverviews(NearestNeighbor)
.write(new Path("xxx"), hconf.value)
// the image distorted output-2
cropRDD.regrid(512, 512).toGeoTiffs()
.filter{
case (_: SpatialKey, null) ⇒ false // remove any null Tiles
case _ ⇒ true
}
.foreach { case (sk: SpatialKey, gt: MultibandGeoTiff) ⇒
val path = new Path(new Path("xxx"), s"${sk.col}_${sk.row}.tif")
MultibandGeoTiff(gt.tile, gt.extent,
tlm.crs, GeoTiffOptions.DEFAULT.copy(compression = DeflateCompression))
.withOverviews(NearestNeighbor)
.write(path, hconf.value)
}
1 reply
Singleband geotiff can be loaded with openlayers, but the MultibandTile geotiff cannot.
when I use gdal(validate_cloud_optimized_geotiff.py) to check it, the following warning like this:
Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.
How can I fix it?
4 replies
I am trying to convert landscan raster cell to polygon using the following code
def rasterToPolygonsT <: Tile: Seq[Polygon] = {
raster.polygonalize().map { case (polygon, tile) => polygon }
}
but not able to find the polygonalize() method, I believe this polygonalize method is defined in the Raster trait, which is the base trait for all raster types in GeoTrellis. but on using in code it is not able to find the definition. I am using import Geotrellis.raster.Raster. Any other import that I need to use, or any other function that can be used.
3 replies
So this is probably a dumb question, but I can't seem to find the answer (likely don't know how to google correctly).
If I were to have a a list of lat lon with an associated Int value. How would I know where to place those values in the IntArrayTile? I know how to calculate the extents, find out what tile they go into to generate the png, but I have no clue how to place those values in the correct positions in the ArrayTile.
5 replies