Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 01 2018 17:29

    metasim on develop

    We've moved! (compare)

  • Apr 24 2018 00:55

    metasim on aws-catalog

    Queryable AWS L8 catalog. (compare)

  • Apr 24 2018 00:26

    metasim on develop

    Feature/python api (#68) * Add… (compare)

  • Apr 24 2018 00:26
    metasim closed #68
  • Apr 24 2018 00:07
    codecov[bot] commented #68
  • Apr 24 2018 00:04
    codecov[bot] commented #68
  • Apr 23 2018 23:41
    codecov[bot] commented #68
  • Apr 23 2018 23:41
    metasim synchronize #68
  • Apr 23 2018 23:41

    metasim on python-api

    Created "experimental" sub-proj… Initial generation of slippy ma… Added auto-generated HTML file … and 3 more (compare)

  • Apr 23 2018 22:08
    codecov[bot] commented #45
  • Apr 23 2018 21:58
    codecov[bot] commented #68
  • Apr 23 2018 21:41

    bguseman on anchor

    Initial skeleton for Python mod… Incremental progress commit. Ported to sbt 1.0.2. Had to aba… and 26 more (compare)

  • Apr 23 2018 21:41
    bguseman synchronize #68
  • Apr 23 2018 21:41

    bguseman on python-api

    Pre-merge cleanup. (compare)

  • Apr 23 2018 21:41

    bguseman on locationtech-base

    (compare)

  • Apr 23 2018 16:31

    metasim on anchor

    (compare)

  • Apr 23 2018 16:31

    metasim on locationtech-base

    (compare)

  • Apr 23 2018 13:37
    metasim commented #68
  • Apr 18 2018 15:14
    bguseman opened #68
  • Apr 18 2018 14:51

    bguseman on python-api

    Minor cleanup. (compare)

Simeon H.K. Fitch
@metasim

Hi @Confman . You can get the "extent" of a GeoTIFF (the four corners) in the native CRS/projection, and then convert them into Lat/Lon.
First know gdalinfo is the fastest way to do this if you just want a quick peek .

You can get the lat/lon info just usiong core GeoTrellis, before dealing with a RasterFrame:

import geotrellis.proj4.LatLng
import geotrellis.raster.io.geotiff.SinglebandGeoTiff

val g = SinglebandGeoTiff("/Users/sfitch/Coding/OSS/locationtech-rasterframes/core/src/test/resources/L8-B2-Elkton-VA.tiff")
val e = g.projectedExtent.reproject(LatLng)
println(e)
println(e.center)

If you already have a raster frame, you can get the (complete) extent from it like this:

val rf: RasterFrame = ...
rf.tileLayerMetadata.merge.extent.reproject(rf.crs, LatLng)

When it comes to elevation, I've never seen that in standard satellite image GeoTIFF... a Digital Elevation Model (DEM) is usually a separate data product.

Hakan
@Confman
Thanks @metasim , I'll check that out. May other type of data reside in a different column, such as tile? Supposedly the following geotiff has weather information inside as well: http://worldclim.org/version2 however I failed to extract that too.
Simeon H.K. Fitch
@metasim
@Confman It is certainly possible that a GeoTIFF can contain any type of data.
If it is a multiband GeoTIFF, those bands will show up as separate columns. See: http://rasterframes.io/creating-rasterframes.html#reading-a-geotiff
If you use spark.read.geotiff.loadRF(...) then the metadata will be stored as a column dictionary/map column. If you want to get at it with just GeoTrellis, then the {Singleband|Multiband}GeoTiff classes have a method called tags where this is stored.
Hakan
@Confman
Thanks for your help @metasim, I very much appreciate it.
Jason T Brown
@vpipkt
@Confman the rasters there are images produced with something like a hillshade technique: http://www.geography.hunter.cuny.edu/~jochen/GTECH361/lectures/lecture11/concepts/Hillshade.htm
So the values in the raster are just brightness of the image; not actual elevations. See something like the SRTM DEM, which should have raster values in meters above sea level. Hope it helps
Hakan
@Confman
Thank you @vpipkt
Christos
@Charmatzis
Hi guys, I think that someone have already mentioned that, but in the https://rasterframes.io documentation I think all the links that go to github pages are broken because they are targeting the develop branch and not the master
Jason T Brown
@vpipkt
good to know! we are undertaking a major re-write of docs to go along with 0.8.0 release.... I'll put this on our list of things to check
alevillafds
@alevillafds
Hi all! I need to calculate a linear reggression with spark ml over a collection of rasterframes, but i don't know if it is possible.
Nicola
@120534
@alevillafds Hi, I'm a user of RasterFrames. Indeed , RasterFrame helps me solve with the same issue you referred. Here is my code dealing making simple Linear Regression between NDVI and Albedo usding RasterFrame and Spark ML, hope it helps.

 //  pull in the target label data.
  val feature = "feature" //ndvi
  val label = "label" //albedo
  // Now, we get a rasterFrame with three columns
  val newRF: DataFrame = albedoRF.select( $"spatial_key", $"ndvi", $"albedo")

  // rename the columns, put ndvi as feature and albedo as label
  val trainRF = newRF.withColumnRenamed("ndvi", "feature")
    .withColumnRenamed("albedo", "label")

  // exploder is used to break the tiles in to single cell data
  import astraea.spark.rasterframes.ml.TileExploder
  val exploder = new TileExploder()

  val colNames = Array("label", "feature")

  val noDataFilter: NoDataFilter = new NoDataFilter()
    .setInputCols(colNames)

  // To "vectorize" the the band columns we use the SparkML `VectorAssembler`
  val assembler: VectorAssembler = new VectorAssembler()
    .setInputCols(Array("feature"))
    .setOutputCol("features")

  // Setting spark ml LinearRegression params
  val classifier: LinearRegression = new LinearRegression()
    .setMaxIter(10)
    .setElasticNetParam(0.8)

  // Assemble the pipeline
  val pipeline: Pipeline = new Pipeline()
    .setStages(Array(exploder, noDataFilter, assembler, classifier))

  val lrModel: PipelineModel = pipeline.fit(trainRF)
  val result = lrModel.transform(trainRF)
  result.show(10)
  val stage3 = lrModel.stages(3).asInstanceOf[LinearRegressionModel]

  println(s"Coefficients: ${stage3.coefficients} Intercept: ${stage3.intercept}")

  // print model summary info
  val trainingSummary = stage3.summary
  println(s"numIterations: ${trainingSummary.totalIterations}")
  println(s"objectiveHistory: [${trainingSummary.objectiveHistory.mkString(",")}]")
  trainingSummary.residuals.show()
  println(s"RMSE: ${trainingSummary.rootMeanSquaredError}")
  println(s"r2: ${trainingSummary.r2}")
alevillafds
@alevillafds
thanks!!! And if i have multiple series of rasters there is a way to calculate linear reggression through multiple series?
y = Multiple rasters, x1 = Multiple rasters
Because i need to generate a raster from a linear regression
alevillafds
@alevillafds
i need to make the regression pixel to pixel
alevillafds
@alevillafds
tile.toArrayDouble() return the pixels in row major?
Jason T Brown
@vpipkt
@alevillafds Our upcoming 0.8.0 release will introduce a concept called catalogs which is a dataframe of pointers to the raster data. Then we use spark.read.raster(catalog=catalog_df,...) to get a dataframe containing all the rows from all the rasters "stacked" as rows.
But for now the answer is yes a regression pixel to pixel is very much possible.
If you use the TileExploder as @120534 demonstrates the resulting schema will put the label and feature pixel values on the same row, ready to go for the rest of spark ML pipeline.
there are some details to consider about whether your label (y) rasters and feature (x) rasters are in the same CRS and on the same gridding.
Jason T Brown
@vpipkt
@alevillafds not sure if that answers your question entirely, feel free to keep asking.
alevillafds
@alevillafds
thank you very much!!!
now i'm doing the regression manual because my formula is: y[1:100] = x1[1:100] + x2[1:100] where [1:100] are a collection of rasters so i calculate n linear reggression for each pixel
alevillafds
@alevillafds
but i dont know how to assemble the result pixels
alevillafds
@alevillafds
@vpipkt when is the 0.8.0 release?
Simeon H.K. Fitch
@metasim

@alevillafds Take a look at this example, and you'll see how you reassemble the result pixels:
https://rasterframes.io/ml/clustering.html#visualizing-results

We're hoping to get the 0.8.0 out by the end of next week. Might be the week after that. It's soon.

Jason T Brown
@vpipkt
@alevillafds things are stabilising a lot on develop branch as 0.8.0 release approaches... so you could try to work with a snapshot :-)
but i am a little confused by the regression formula above y[1:100] = x1[1:100] + x2[1:100]
and the comment that the [1:100] is a collection of rasters
you need to calculate n linear regressions for each pixel. Does that mean at each pixel location there are (in this example) 99 different regressions to solve. e.g. y[1] = b1[1] * x1[1] + b2[1]*x2[1] and separately y[2] = b1[2] * x1[2] + b2[2]*x2[2]and y[2] = b1[2] * x1[2] + b2[2]*x2[2] ... y[100] = b1[100] * x1[100] + b2[100]*x2[100] ?
alevillafds
@alevillafds

well, i'm trying to simulate the linear regression in r:
s <- stack(RASTERS1, RASTERS2, RASTERS3, RASTERS4)
fun <- function(x) {if (is.na(x[1])) { NA } else {lm(x[1:100] ~ x[101:200] + x[201:300] + x[301:400])$coefficients[1]}
rasterResult <- calc(s, fun)

and rasterResult is a raster that corresponds to coefficients[1]

Now i'm calculating linear reggresion from pixel 1, pixel 2, pixel 3, etc in all timeseries, so i build the dataframe manually. In each dataframe, each column corresponds to the same pixel in the timeseries.This is very slow because i need to calculare 200000+- linear regression (each raster have 200000 pixels)

Sorry for my english, i'm spanish

I think I'm going to wait for version 0.8.0 to come out because I don't know if I'm doing the calculation correctly, having to align the pixels manually
Jason T Brown
@vpipkt
@alevillafds your english is >>> my spanish for sure!
Jason T Brown
@vpipkt
and the R i can read pretty well. In that example what type are RASTERS1, RASTERS2,... are they 2d matrices? data.frame? And if 2d do they have 100 elements? Like you are trying to predict something like lm(RASTER1 ~ RASTER2 + RASTER3 + RASTER4 ) ?
And computing a different linear regression per pixel (the 200,000 regressions) is a different problem than computing the linear regression considering all pixels as observations
RasterFrames is good at this latter case; computing (ml model / linear regression) considering all pixels as observations.
Jason T Brown
@vpipkt
Here is a recent build of docs showing a supervised learning example. We build the label (or Y) raster from some vector data, but the ideas are simpler if you already have the label raster (and it is in the same CRS and gridding as your features/X's)
https://992-174014757-gh.circle-artifacts.com/0/rf-site/supervised-learning.html
:point_up: that is with pre-release 0.8.0
Jason T Brown
@vpipkt
Just about all set for 0.8.0 release: https://github.com/locationtech/rasterframes/milestone/1
Simeon H.K. Fitch
@metasim
:tada:
Simeon H.K. Fitch
@metasim
@/all In preparation for 0.8.0, this channel is going to be retired and subsequent messaging placed under the LocationTech banner at https://gitter.im/locationtech/rasterframes. Thanks to all who have contributed to the dialogue thus far!
Simeon H.K. Fitch
@metasim
Hope to see you all in our new forum!
Jason T Brown
@vpipkt
:wave:
Rob Knapen
@robknapen
🎉👋🏼
MiguelNOX
@MiguelNOX

Hello! i have a problem reading a raster file through RasterFrames, i hope someone can help me out...

The file is in ENVI format, but i've also tried to read it after converting it to TIF. I'm trying to read it from a local computer (not hosted in cloud services as the example of the RF homepage).

BTW, i've read the doc page's example without any trouble.

image.png
image.png
I've been able to read the file with gdal and w/o RF, however, i need to work with it through pyspark (ie RasterFrames) due to the big amount of data
Jason T Brown
@vpipkt
@MiguelNOX thanks for asking the same on our LocationTech channel .... https://gitter.im/locationtech/rasterframes we'll continue the conversation over there