locationtech/rasterframes

Geospatial raster support for Spark DataFrames

wxmimperio

@imperio-wxm

Whether the parameters query and paginationToken in StacApi are use cases, I didn't find them in the test case and don't know how to use these two parameters.

implicit val searchFiltersDecoder: Decoder[SearchFilters] = { c =>
    for {
      bbox              <- c.downField("bbox").as[Option[Bbox]]
      datetime          <- c.downField("datetime").as[Option[TemporalExtent]]
      intersects        <- c.downField("intersects").as[Option[Geometry]]
      collectionsOption <- c.downField("collections").as[Option[List[String]]]
      itemsOption       <- c.downField("ids").as[Option[List[String]]]
      limit             <- c.downField("limit").as[Option[NonNegInt]]
      query             <- c.get[Option[Map[String, List[Query]]]]("query")
      paginationToken   <- c.get[Option[PaginationToken]]("next")
    } yield {
      SearchFilters(
        bbox,
        datetime,
        intersects,
        collectionsOption getOrElse Nil,
        itemsOption getOrElse Nil,
        limit,
        query getOrElse Map.empty,
        paginationToken
      )
    }
  }

Grigory

@pomadchin

hey @imperio-wxm that’s an outdated codec, RF needs upgrade up to https://github.com/azavea/stac4s/releases/tag/v0.8.0

not sure what your question is though...

wxmimperio

@imperio-wxm

@pomadchin hi,I upgrade version to 0.8.0, But I still don't know what value should be passed in this place. I want to filter out the items whose title is Band 1 - Coastal aerosol - 60m on the second page.Is there some test case to cover, maybe because I'm not familiar with stac4s.

val results = spark
        .read
        .stacApi(
          "https://planetarycomputer.microsoft.com/api/stac/v1/",
          filters = SearchFilters(
            collections = List("sentinel-2-l2a"),
            query = Map("title" -> List(Equals(Json.fromString("Band 1 - Coastal aerosol - 60m")))),
            paginationBody = JsonObject.fromMap(Map("page" -> Json.fromString("2")))
          )
        )
        .load.limit(10)

Grigory

@pomadchin

hey @imperio-wxm I dont think MSFT PC API support the fragmets extension API; but if it doest htat’s allright

the pagination body in this case should be smth like "token": "next:S2A_MSIL2A_20220411T101601_R065_T45XWK_20220412T055304”

check out the STAC API Spec for more details

do you have a working JSON body you want to send?

wxmimperio

@imperio-wxm

hi @pomadchin What does the query parameter mean and what should I pass in? My question is if I want something similar to STAC Search, how should I use the rasterframes code to implement it?

Grigory

@pomadchin

@imperio-wxm that’s for the older query fragment extension https://github.com/radiantearth/stac-api-spec/tree/master/fragments/query

STAC4S does not support https://github.com/radiantearth/stac-api-spec/tree/master/fragments/filter but any contributions are welcome

wxmimperio

@imperio-wxm

How to write a full resolution GeoTIFF? I saw the restriction on the geotiff writer on the official website, which needs to read all the data to the driver, but this will cause oom. So you need to use withDimension to limit the output resolution. But what is the correct operation if I want to write out a full resolution tiff?

// For example, I perform log calculation on the band5 file of landsat8
// And then output the calculated image, the image resolution is 7581x7721.
val out = Paths.get("target", "band5.tif")
      noException shouldBe thrownBy {
        df.select(rf_log(col("band5")))
          .write.geotiff
          .withCRS(LatLng)
          //.withDimensions(1024, 1024)
          .save(out.toString)
      }

wxmimperio

@imperio-wxm

Why is Ndvi's calculation different from QGIS?

My QGIS formula:

 ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" - "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )  /  ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" + "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )

My spark code:

val ndvi = df.withColumn("ndvi", rf_normalized_difference(col("band5"), col("band4")))
ndvi.printSchema()
val out = Paths.get("target", "spark_ndvi.tif")
noException shouldBe thrownBy {
ndvi.write.geotiff
    .withCRS(LatLng)
    .withDimensions(4096, 4096)
    .save(out.toString)
}

qgis ndvi

spark ndvi

Grigory

@pomadchin

hey @imperio-wxm could be because of a cellType, try to convert band5 and band4 into doubles before computing ndvi

wxmimperio

@imperio-wxm

@pomadchin hi, it not work my code:

 val ndvi = df.withColumn("ndvi", rf_normalized_difference(rf_convert_cell_type(col("band5"), DoubleCellType), rf_convert_cell_type(col("band4"), DoubleCellType)))

How to write a full resolution GeoTIFF? I saw the restriction on the geotiff writer on the official website, which needs to read all the data to the driver, but this will cause oom. So you need to use withDimension to limit the output resolution. But what is the correct operation if I want to write out a full resolution tiff?
// For example, I perform log calculation on the band5 file of landsat8
// And then output the calculated image, the image resolution is 7581x7721.
val out = Paths.get("target", "band5.tif")
      noException shouldBe thrownBy {
        df.select(rf_log(col("band5")))
          .write.geotiff
          .withCRS(LatLng)
          //.withDimensions(1024, 1024)
          .save(out.toString)
      }

@pomadchin hi, Is there any solution for writing out at full resolution?

Grigory

@pomadchin

@imperio-wxm the only solution is to give driver more mem

wxmimperio

@imperio-wxm

@pomadchin thanks

@pomadchin it is not working. Code like val ndvi = df.withColumn("ndvi", rf_normalized_difference(rf_convert_cell_type(col("band5"), DoubleCellType)

Grigory

@pomadchin

@imperio-wxm what do you mean by ‘not working’?

wxmimperio

@imperio-wxm

@pomadchin After converting the celltype to double type, the calculation of ndvi is still different from that of qgis, which is not a problem of celltype.

wxmimperio

@imperio-wxm

Why is Ndvi's calculation different from QGIS?

My QGIS formula:

 ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" - "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )  /  ( "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B5@1" + "LC08_L2SP_119038_20210623_20210630_02_T1_SR_B4@1" )

My spark code:

val ndvi = df.withColumn("ndvi", rf_normalized_difference(col("band5"), col("band4")))
ndvi.printSchema()
val out = Paths.get("target", "spark_ndvi.tif")
noException shouldBe thrownBy {
ndvi.write.geotiff
    .withCRS(LatLng)
    .withDimensions(4096, 4096)
    .save(out.toString)
}

qgis ndvi

spark ndvi

@pomadchin Hi, Is there any answer to this question?

Grigory

@pomadchin

hey @imperio-wxm

I think at this point, the help can only be done by inspecting those rasters only

could you share it mb?

Simeon H.K. Fitch

@metasim

@imperio-wxm What happens if you remove the withCRS(LatLng)?

The fact that there's a maximum NDVI value of 75 is just bizarre.

Maybe try reading and then immediately writing a single band in RF and comparing that with gdalinfo/qgis.

wxmimperio

@imperio-wxm

@metasim Hi, If i remove withCRS get an error:

Caused by: java.lang.IllegalArgumentException: A destination CRS must be provided
    at org.locationtech.rasterframes.datasource.geotiff.GeoTiffDataSource.$anonfun$createRelation$7(GeoTiffDataSource.scala:73)
    at scala.Option.getOrElse(Option.scala:189)
    at org.locationtech.rasterframes.datasource.geotiff.GeoTiffDataSource.createRelation(GeoTiffDataSource.scala:73)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)

3 replies

zoneyuan

@zoneyuan

Hey can someone help me? I do installed pyrasterframe successfully, but when I try the example, I cant read raster correctly

seems like spark.read doesn't include raster, what should i do?

Simeon H.K. Fitch

@metasim

@zoneyuan The column df.tile doesn't exist. Print the schema to discover what columns are available for that image file.

zoneyuan

@zoneyuan

@metasim get it, thanks :)

kembles5

@kembles5

Hello, i'm attempting to use pyrasterframes from the pyspark shell. I set the py-files config to the zip file i downloaded. I've tried zip files for both 2.11 (which fails b/c I'm using spark 3) and 2.12 which fails to find the geotrellis dependency. Digging a little deeper and it looks as if geotrellis does not run on spark 3. If that's the case, does rasterframes also not run on spark 3?

jpolchlo

@jpolchlo

@kembles5 Geotrellis does run on Spark 3. That's not to say that I know what problem you're running into, but GT 3.6.0 and up are spark 3 compatible.

You may want to check which version is included in the zip file you want to use.

I've been running rasterframes 0.10.1 (in Scala, though) in spark 3.1.2 without a problem.

kembles5

@kembles5

@jpolchlo good to hear that it runs on spark 3. I'm getting the following module not found: org.locationtech.geotrellis#geotrellis-spark_2.12;3.6.1-SNAPSHOT

which may be related to an issue on the geotrellis side

glancing at the geotrellis gitter and there's mention today of a build/dependency issue

kembles5

@kembles5

Also, get this error: SERVER ERROR: Bad Gateway url=https://dl.bintray.com/spark-packages/maven/org/locationtech/geotrellis/geotrellis-spark_2.12/3.6.1-SNAPSHOT/maven-metadata.xml.

which i do not get if I use the 2.12-0.9.1 zip file

sorry, meant 2.11-0.9.1

jpolchlo

@jpolchlo

@kembles5 OK, that makes sense. Bintray shut down, so our snapshots are being hosted elsewhere. I think https://repo.eclipse.org/content/repositories/geotrellis-snapshots/org/locationtech/geotrellis/ ? I'm not 100% on how to use this information to solve your problem. Hopefully one of the rasterframes engineers can speak to that.

In the past I've used the --repositories flag to spark-submit to download packages from nonstandard locations. Perhaps there's a way to do that with pyspark?

kembles5

@kembles5

@jpolchlo thank you. i'll see if i can add a --repository flag. fwiw, here's the command i'm running:

pyspark --archives hub_env.tar.gz --py-files pyrasterframes_2.12-0.10.0-python.zip --packages org.locationtech.rasterframes:rasterframes_2.12:0.10.0,org.locationtech.rasterframes:pyrasterframes_2.12:0.10.0,org.locationtech.rasterframes:rasterframes-datasource_2.12:0.10.0 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrator=org.locationtech.rasterframes.util.RFKryoRegistrator --conf spark.kryoserializer.buffer.max=500m

Grigory

@pomadchin

hey @kembles5 I am not sure why do you need a snapshot, the most recent gt release is 3.6.2 and the most recent RF release is is 0.10.1 which depends on GT 3.6.1 transitively;

RF of versions prior to 0.10.0 are not compatible with Spark 3.x

if you still need GT snapshots than they are available on the maven nexus i.e.: https://oss.sonatype.org/content/repositories/snapshots/org/locationtech/geotrellis/geotrellis-spark_2.12/

Check out the GT README badges: https://github.com/locationtech/geotrellis#geotrellis

Where communities thrive

People

Repo info

Activity