GeoTrellis is a geographic data processing engine for high performance applications.
@pomadchin Hi,my code is sample:
val sourceRDD: RDD[RasterSource] = sc.parallelize(files, 50).filter(url => url.endsWith(".tiff")).map(uri => {
GeoTiffRasterSource(uri): RasterSource
})
val summary = RasterSummary.fromRDD(sourceRDD)
val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
val contextRDD = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = Some(summary))
contextRDD.foreach { case (key, tile) =>
println(tile.size)
}
I increasing paralelism like this, but there are only 11 tasks actually read in foreach.
sc.parallelize(files, 50)
I added logs and outputs to the source code of S3. Obviously, there are a lot of the same range being read repeatedly. There are too many logs and I only intercept a part of them.
the problem here is that you have a layout, right? and you want to tile the scene accrding to some layout definition
but what is the tiling scheme of every TIFF you have?
@pomadchin
HI, What is the index order of MultibandTile bands?
// I read three tiffs:
// LC09_L2SP_148033_20220512_20220514_02_T1_SR_B2.TIF
// LC09_L2SP_148033_20220512_20220514_02_T1_SR_B3.TIF
// LC09_L2SP_148033_20220512_20220514_02_T1_SR_B4.TIF
val sourceRDD: RDD[RasterSource] = sc.parallelize(files, 10)
.map(uri => {
GeoTiffRasterSource(uri): RasterSource
}).cache()
multibandTile.band(0) = SR_B2,multibandTile.band(1) = SR_B3,multibandTile.band(2) = SR_B4?
How to accurately select the band I want by index, what is the mapping relationship between index and band?
val rastersources = layer_filenames.mapPartitions(partition => { S3Utils.configureGeotrellis(s3confBroadcast.value); partition.map(x => GeoTiffRasterSource(x))})
val epsg4326 = CRS.fromEpsgCode(4326)
val rastersources_webmercator = rastersources.map(x => x.reproject(epsg4326, method = Max).reproject(WebMercator, method = Max))
val summary = RasterSummary.fromRDD(rastersources_webmercator)
val layoutScheme = ZoomedLayoutScheme(WebMercator)
val LayoutLevel(maxZoom, layoutDefinition) = summary.levelFor(layoutScheme)
val layer_webmercator = RasterSourceRDD.tiledLayerRDD(rastersources_webmercator, layoutDefinition, KeyExtractor.spatialKeyExtractor, rasterSummary = Some(summary))
val layer_webmercator_singleband = ContextRDD(layer_webmercator.map(x => (x._1,x._2.band(0))) , layer_webmercator.metadata)
Pyramid.upLevels(lyr, layoutScheme, maxZoom, Max) { (rdd, z) =>
val layerId = LayerId(layerName, z)
if(store.layerExists(layerId)) deleter.delete(layerId)
writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}
RasterSource(tiff).tileToLayout(...).read(key)
<- the min example
val sourceRdd: RDD[RasterSource] = sc.parallelize(
Seq("LC09_L2SP_148033_20220512_20220514_02_T1_SR_B4.TIF", "LC09_L2SP_148033_20220512_20220514_02_T1_SR_B5.TIF"),
2
).map(uri => {
GeoTiffRasterSource(uri): RasterSource
}).cache()
val summary: RasterSummary[Unit] = RasterSummary.fromRDD(sourceRdd)
val LayoutLevel(_, layout) = summary.levelFor(FloatingLayoutScheme(256))
val contextRDD = RasterSourceRDD.tiledLayerRDD(sourceRdd, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = Some(summary)).persist(StorageLevel.DISK_ONLY)
contextRDD.map {
case (key, tile) =>
// bandCount == 2
println(tile.bandCount)
// B4 or B5 ?
println(tile.band(0))
}.collect()