Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:07
    metasim commented #486
  • Aug 18 19:26
    BKDaugherty commented #486
  • Aug 15 15:30
    BKDaugherty starred locationtech/rasterframes
  • Aug 10 06:19
    spacefan starred locationtech/rasterframes
  • Jul 27 13:20
    pomadchin commented #589
  • Jul 26 17:05
    echeipesh opened #589
  • Jul 26 17:05
    echeipesh labeled #589
  • Jul 16 22:57
  • Jul 16 22:57
  • Jul 12 13:17
    pomadchin commented #587
  • Jul 12 03:50
    echeipesh commented #587
  • Jul 10 20:26
    pomadchin commented #587
  • Jul 06 01:01
    echeipesh synchronize #587
  • Jul 06 01:00

    echeipesh on spark-3.2

    Fix UDF style Aggregates (compare)

  • Jul 05 13:19
    imperio-wxm opened #588
  • Jun 30 05:02
    echeipesh opened #587
  • Jun 30 05:01

    echeipesh on spark-3.2

    bumped dev version Spark 3.2 Lets get it compiled… withNewChildrenInternal and 2 more (compare)

  • Jun 18 00:19
  • Jun 14 13:02
    dependabot[bot] labeled #586
  • Jun 14 13:02
    dependabot[bot] opened #586
wxmimperio
@imperio-wxm
@pomadchin Hi, I run tests in intelij IDEA.
Grigory
@pomadchin
@imperio-wxm all right, indeed there is a bug / feature in how idea resolves dependencies for some reason; exclude in idea the old paranamer dependency
Or use SBT to run tests
wxmimperio
@imperio-wxm
@pomadchin Hi, what old paranamer dependencies need to be excluded?
Grigory
@pomadchin
If you’ll look into the project deps, there will be two com.thoughtworks.paranamer deps in the project; just exclude the older one
That’s a great question how the old one gets into the idea project deps tree
If you can’t figure it out just use a plain sbt to run tests
wxmimperio
@imperio-wxm
@pomadchin thanks, exclude version 2.3, it works.
Grigory
@pomadchin
@imperio-wxm :+1: nice
Simeon H.K. Fitch
@metasim
@imperio-wxm Warning: if you reload the project in IntelliJ, you'll have to do the manual exclude again :/
(I think it has to do with IntelliJ merging the project dependencies with sbt plugin dependencies)
Grigory
@pomadchin
wow
wxmimperio
@imperio-wxm
Whether the parameters query and paginationToken in StacApi are use cases, I didn't find them in the test case and don't know how to use these two parameters.
implicit val searchFiltersDecoder: Decoder[SearchFilters] = { c =>
    for {
      bbox              <- c.downField("bbox").as[Option[Bbox]]
      datetime          <- c.downField("datetime").as[Option[TemporalExtent]]
      intersects        <- c.downField("intersects").as[Option[Geometry]]
      collectionsOption <- c.downField("collections").as[Option[List[String]]]
      itemsOption       <- c.downField("ids").as[Option[List[String]]]
      limit             <- c.downField("limit").as[Option[NonNegInt]]
      query             <- c.get[Option[Map[String, List[Query]]]]("query")
      paginationToken   <- c.get[Option[PaginationToken]]("next")
    } yield {
      SearchFilters(
        bbox,
        datetime,
        intersects,
        collectionsOption getOrElse Nil,
        itemsOption getOrElse Nil,
        limit,
        query getOrElse Map.empty,
        paginationToken
      )
    }
  }
Grigory
@pomadchin
hey @imperio-wxm that’s an outdated codec, RF needs upgrade up to https://github.com/azavea/stac4s/releases/tag/v0.8.0
not sure what your question is though...
wxmimperio
@imperio-wxm
@pomadchin hi,I upgrade version to 0.8.0, But I still don't know what value should be passed in this place. I want to filter out the items whose title is Band 1 - Coastal aerosol - 60m on the second page.Is there some test case to cover, maybe because I'm not familiar with stac4s.
val results = spark
        .read
        .stacApi(
          "https://planetarycomputer.microsoft.com/api/stac/v1/",
          filters = SearchFilters(
            collections = List("sentinel-2-l2a"),
            query = Map("title" -> List(Equals(Json.fromString("Band 1 - Coastal aerosol - 60m")))),
            paginationBody = JsonObject.fromMap(Map("page" -> Json.fromString("2")))
          )
        )
        .load.limit(10)
Grigory
@pomadchin
hey @imperio-wxm I dont think MSFT PC API support the fragmets extension API; but if it doest htat’s allright
the pagination body in this case should be smth like "token": "next:S2A_MSIL2A_20220411T101601_R065_T45XWK_20220412T055304”
check out the STAC API Spec for more details
do you have a working JSON body you want to send?
wxmimperio
@imperio-wxm
hi @pomadchin What does the query parameter mean and what should I pass in? My question is if I want something similar to STAC Search, how should I use the rasterframes code to implement it?
image.png
Grigory
@pomadchin
@imperio-wxm that’s for the older query fragment extension https://github.com/radiantearth/stac-api-spec/tree/master/fragments/query
STAC4S does not support https://github.com/radiantearth/stac-api-spec/tree/master/fragments/filter but any contributions are welcome
wxmimperio
@imperio-wxm
How to write a full resolution GeoTIFF? I saw the restriction on the geotiff writer on the official website, which needs to read all the data to the driver, but this will cause oom. So you need to use withDimension to limit the output resolution. But what is the correct operation if I want to write out a full resolution tiff?
// For example, I perform log calculation on the band5 file of landsat8
// And then output the calculated image, the image resolution is 7581x7721.
val out = Paths.get("target", "band5.tif")
      noException shouldBe thrownBy {
        df.select(rf_log(col("band5")))
          .write.geotiff
          .withCRS(LatLng)
          //.withDimensions(1024, 1024)
          .save(out.toString)
      }
wxmimperio
@imperio-wxm

Why is Ndvi's calculation different from QGIS?

My QGIS formula:

 ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" - "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )  /  ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" + "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )

My spark code:

val ndvi = df.withColumn("ndvi", rf_normalized_difference(col("band5"), col("band4")))
ndvi.printSchema()
val out = Paths.get("target", "spark_ndvi.tif")
noException shouldBe thrownBy {
ndvi.write.geotiff
    .withCRS(LatLng)
    .withDimensions(4096, 4096)
    .save(out.toString)
}

qgis ndvi
image.png
image.png

spark ndvi
image.png
image.png

Grigory
@pomadchin
hey @imperio-wxm could be because of a cellType, try to convert band5 and band4 into doubles before computing ndvi
wxmimperio
@imperio-wxm
@pomadchin hi, it not work my code:
 val ndvi = df.withColumn("ndvi", rf_normalized_difference(rf_convert_cell_type(col("band5"), DoubleCellType), rf_convert_cell_type(col("band4"), DoubleCellType)))

How to write a full resolution GeoTIFF? I saw the restriction on the geotiff writer on the official website, which needs to read all the data to the driver, but this will cause oom. So you need to use withDimension to limit the output resolution. But what is the correct operation if I want to write out a full resolution tiff?

// For example, I perform log calculation on the band5 file of landsat8
// And then output the calculated image, the image resolution is 7581x7721.
val out = Paths.get("target", "band5.tif")
      noException shouldBe thrownBy {
        df.select(rf_log(col("band5")))
          .write.geotiff
          .withCRS(LatLng)
          //.withDimensions(1024, 1024)
          .save(out.toString)
      }

@pomadchin hi, Is there any solution for writing out at full resolution?

Grigory
@pomadchin
@imperio-wxm the only solution is to give driver more mem
wxmimperio
@imperio-wxm
@pomadchin thanks
@pomadchin it is not working. Code like val ndvi = df.withColumn("ndvi", rf_normalized_difference(rf_convert_cell_type(col("band5"), DoubleCellType)
Grigory
@pomadchin
@imperio-wxm what do you mean by ‘not working’?
wxmimperio
@imperio-wxm
@pomadchin After converting the celltype to double type, the calculation of ndvi is still different from that of qgis, which is not a problem of celltype.
wxmimperio
@imperio-wxm

Why is Ndvi's calculation different from QGIS?

My QGIS formula:

 ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" - "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )  /  ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" + "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )

My spark code:

val ndvi = df.withColumn("ndvi", rf_normalized_difference(col("band5"), col("band4")))
ndvi.printSchema()
val out = Paths.get("target", "spark_ndvi.tif")
noException shouldBe thrownBy {
ndvi.write.geotiff
    .withCRS(LatLng)
    .withDimensions(4096, 4096)
    .save(out.toString)
}

qgis ndvi
image.png
image.png

spark ndvi
image.png
image.png

@pomadchin Hi, Is there any answer to this question?

Grigory
@pomadchin

hey @imperio-wxm

I think at this point, the help can only be done by inspecting those rasters only

could you share it mb?
Simeon H.K. Fitch
@metasim
@imperio-wxm What happens if you remove the withCRS(LatLng)?
The fact that there's a maximum NDVI value of 75 is just bizarre.
Maybe try reading and then immediately writing a single band in RF and comparing that with gdalinfo/qgis.
wxmimperio
@imperio-wxm
@metasim Hi, If i remove withCRS get an error:
Caused by: java.lang.IllegalArgumentException: A destination CRS must be provided
    at org.locationtech.rasterframes.datasource.geotiff.GeoTiffDataSource.$anonfun$createRelation$7(GeoTiffDataSource.scala:73)
    at scala.Option.getOrElse(Option.scala:189)
    at org.locationtech.rasterframes.datasource.geotiff.GeoTiffDataSource.createRelation(GeoTiffDataSource.scala:73)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
3 replies
zoneyuan
@zoneyuan
Hey can someone help me? I do installed pyrasterframe successfully, but when I try the example, I cant read raster correctly
image.png
image.png
seems like spark.read doesn't include raster, what should i do?
Simeon H.K. Fitch
@metasim
@zoneyuan The column df.tile doesn't exist. Print the schema to discover what columns are available for that image file.
zoneyuan
@zoneyuan
@metasim get it, thanks :)
kembles5
@kembles5
Hello, i'm attempting to use pyrasterframes from the pyspark shell. I set the py-files config to the zip file i downloaded. I've tried zip files for both 2.11 (which fails b/c I'm using spark 3) and 2.12 which fails to find the geotrellis dependency. Digging a little deeper and it looks as if geotrellis does not run on spark 3. If that's the case, does rasterframes also not run on spark 3?
jpolchlo
@jpolchlo
@kembles5 Geotrellis does run on Spark 3. That's not to say that I know what problem you're running into, but GT 3.6.0 and up are spark 3 compatible.
You may want to check which version is included in the zip file you want to use.
I've been running rasterframes 0.10.1 (in Scala, though) in spark 3.1.2 without a problem.