Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 16 02:22

    pomadchin on npm_and_yarn

    (compare)

  • Jan 16 02:21

    pomadchin on master

    Bump shelljs from 0.8.4 to 0.8.… (compare)

  • Jan 16 02:21
    pomadchin closed #3443
  • Jan 15 07:02
    dependabot[bot] labeled #3443
  • Jan 15 07:02
    dependabot[bot] opened #3443
  • Jan 15 07:02

    dependabot[bot] on npm_and_yarn

    Bump shelljs from 0.8.4 to 0.8.… (compare)

  • Jan 15 01:37
    pomadchin labeled #3442
  • Jan 13 10:02
    santocp94 opened #3442
  • Jan 12 20:12
    pomadchin commented #3437
  • Jan 12 20:12
    pomadchin unlabeled #3437
  • Jan 12 20:12
    pomadchin edited #3437
  • Jan 12 20:04
    pomadchin commented #2893
  • Jan 12 19:19
    waveskimmer commented #2893
  • Dec 25 2021 19:45
    pomadchin closed #2615
  • Dec 25 2021 19:45
    pomadchin commented #2615
  • Dec 25 2021 19:42

    pomadchin on master

    Update README.md (compare)

  • Dec 25 2021 18:58

    pomadchin on master

    Fix CI release (compare)

  • Dec 25 2021 18:36

    pomadchin on master

    Automate release process throug… (compare)

  • Dec 25 2021 18:36
    pomadchin closed #3440
  • Dec 25 2021 18:36
    pomadchin closed #3140
yang162132
@yang162132
And by the way is there a function to make two layers with a band to one layer with two bands
林海听枫
@stjimreal
Hello! How could I reproject a MultibandTileLayerRDD? When trying geotrellis-landsat-tutorial, I simply migrate to a newer "org.locationtech.geotrellis" %% "geotrellis-spark" % "3.5.2", however it may be incompatible to reproject a RasterRDD.
And how to read from a pyramid type datasat using "pipeline json string"?
Grigory
@pomadchin
hey @stjimreal this tutorial is just very outdated :/ probably imports are messed up
林海听枫
@stjimreal
@pomadchin I did make it to update the APIs to GT3.x. Well, I just connot fetch out the way to reproject a MultibandTileLayerRDD. Is it because the lack of some other packages?
林海听枫
@stjimreal
Here is my ingest code, it is for a multiband geotiff. It works, but how to reproject a MultibandTileLayerRDD?
def run(implicit sc: SparkContext): Unit = {
    val inputRdd: RDD[RasterSource] = {
    sc.parallelize(inputPath, inputPath.length)
      .map(uri => RasterSource(uri))
      .cache()
    }
    val layoutScheme = ZoomedLayoutScheme(WebMercator, tileSize = 256)
    val summary = RasterSummary.fromRDD(inputRdd)
    val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
    val tiled:MultibandTileLayerRDD[SpatialKey] = RasterSourceRDD
        .tiledLayerRDD(inputRdd, layout)
// We need to reproject the tiles to WebMercator
//    TileRDDReproject(tiled, WebMercator, layoutScheme, Some(Bilinear))
    val attributeStore = FileAttributeStore(outputPath)
    val writer = FileLayerWriter(attributeStore)
    Pyramid.fromLayerRDD(tiled, Some(zoom), Some(0)).write("landsat",
      writer,
      ZCurveKeyIndexMethod)
  }
Grigory
@pomadchin
Hey! Didn’t have a chance to look into your question but will do so shortly
林海听枫
@stjimreal

Hey! Didn’t have a chance to look into your question but will do so shortly

@pomadchin Thx! Would you like to conclude all the ways to ingest/output geotiffs and do ETL things? According to my exploring on the internet, there may have three ways like 1) RasterLayerRDD func 2) MAML 3) pipeline DSL. Where could I find a new DEMO with GT3.x on some web attributes?

Grigory
@pomadchin

hey @stjimreal;

we wanted to avoid terms ETL and to provide a good syntax so it is easy to build own pipelines.
RasterSources syntax and DSL is some sort of a nice replacement to the ETL.

However, Pipeline project is the closes to the old ETL one.

MAML is about lazy raster and rdds transformations

GT server uses GT 3.6 https://github.com/geotrellis/geotrellis-server/tree/main/stac-example

Jan Van den bosch
@bossie
good afternoon my friends
I'm instantiating Geotiff*RasterSources with an s3:// URI; that used to work fine but now I have to support reading geotiffs in buckets in different regions at the same time. How do I tell the underlying S3Client with which region it should communicate?
Grigory
@pomadchin
hey @bossie you can configure client via the S3ClientProducer.set method
Jan Van den bosch
@bossie
yes but that singleton is not going to work in this multithreaded application
Grigory
@pomadchin
@bossie oh fair enough hm
well i’d say that you could have different prupose clients in this case for different regions
allocating a client per thread is certainly an option but very slow one
:D
could you mock up some code to demonstrate the API usage and what you want (ideally) to achieve?
Grigory
@pomadchin
hey @chdsb could you clear it up: what value and when is different? you mean after the ingestion you’re reading a chip via the colleciton reader and the min value is different than in the source? If I understood it correctly than it is due to reprojection + resampling + regridding. Since the ingestion was done in your case into the GeoTrellis layers. That is a precision issue
@chdsb I think it just happened so (: it could also not match
Grigory
@pomadchin
if you want a complete match to the source - I would recommend to consider operating with unchanged rasters via the RasterSource API
@chdsb answering the second question: how do you read sources into the RDD at the moment?
chdsb
@chdsb
@pomadchin hadoopGeoTiffRDD
Grigory
@pomadchin

@chdsb okay, and you would like to use smth like sc.hadoopTemporalGeoTiffRDD but to pass time somehow different rather than setting TIFF tags.

You could use HadoopGeoTiffRDD.singleband directly:

Where I = ProjectedExtent and K = TemporalProjectedExtent; the function you’re interested in is uriToKey: (URI, I) => Kyou can get the temporal information from the path (that used to be one of the ways temporal metadata could be encoded) of the tiff and add it to the SpatialComponent

林海听枫
@stjimreal
@pomadchin great! I will check it.
Grigory
@pomadchin
@yang162132 yea you’re right that’s a bug
a typo
(:
tech11-much
@tech11-much
when the tif is bigger than 800mb, I just can't get the Rdd of the tif. Please tell me how to use the lib.
image.png
Grigory
@pomadchin
Hey @tech11-much , what is the error you are seeing?
tech11-much
@tech11-much
@pomadchin this function hadoopgeotiffrdd() can't read big tiff file
Grigory
@pomadchin
@tech11-much how is that so? what means can’t read? throws an error oh what happens?
tech11-much
@tech11-much
image.png
Grigory
@pomadchin
how large is this tiff?
tech11-much
@tech11-much
2 gb
a 1.5gb makes a different error
Grigory
@pomadchin
2gb is a tiny tiff
I guess, if it is not very well compressed and it is not a bit cell typed tiff

what errors do you have? but typically that means that or you’re pointing to a wrong file or smth else is happening

What gt version are you also on?

tech11-much
@tech11-much
image.png
maybe the tif is wrong
Grigory
@pomadchin
@tech11-much well first of all the GT version is too old, try with 3.6.0
another one is for sure can be related to the TIFF structure, you can try to access it using GDAL RasterSourc
jterry64
@jterry64

I'm running into the issue discussed in this bug: https://github.com/locationtech/geotrellis/issues/3184#issuecomment-595326991
The same read issue happens to me randomly (though not frequently), but I'm not sure I totally understand the resolution. I'm using geotrellis 3.5.2, GDAL 3.1.2, and looks like the GDAL warp jar is 1.1.1. So it seems like it should've had the fix.

I'm also doing this in an environment where it's getting data from a S3 bucket using requester pays, but it definitely has AWS credentials and seems to read fine 99% of the time. Could this be related to the parallel reads like discussed in the bug?

jterry64
@jterry64
And we also run this process in the AWS account that owns the S3 bucket, and I can't seem to find the same errors there
Grigory
@pomadchin
hey @jterry64 what error exactly you have?
I remember this one, was a tricky bug

but what really was in this issue is that GDAL / S3 timetouts reads sometimes due to load / etc
Increase the number of attempts;

the error code for increased amount of attempts is 100