Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:42
    CloudNiner synchronize #3200
  • 14:35
    pomadchin edited #2893
  • 14:33
    pomadchin edited #2893
  • 14:32
    pomadchin edited #2893
  • 14:31
    pomadchin edited #2893
  • 14:31
    pomadchin edited #2893
  • 14:27
    pomadchin edited #2893
  • 14:27
    pomadchin edited #2893
  • 13:52
    echeipesh opened #3217
  • Mar 31 21:27
    CloudNiner synchronize #3200
  • Mar 31 21:11
    CloudNiner commented #3216
  • Mar 31 20:57
    CloudNiner synchronize #3200
  • Mar 31 19:15
    CloudNiner synchronize #3200
  • Mar 31 19:07
    CloudNiner synchronize #3200
  • Mar 31 19:03
    CloudNiner synchronize #3200
  • Mar 31 18:34
    CloudNiner synchronize #3200
  • Mar 31 16:46

    pomadchin on master

    Return back compatible Jackson … (compare)

  • Mar 31 16:46
    pomadchin closed #3215
  • Mar 31 16:37
    moradology labeled #3216
  • Mar 31 16:37
    moradology opened #3216
Grigory
@pomadchin
There it still a problem how would you collect metadata of multiple raster sources
Simeon H.K. Fitch
@metasim
Did you do some experimental work on it?... seem to remember some code somewhere ages ago....
Simeon H.K. Fitch
@metasim
where you were writing out debugging vrt files? or am I making that up.
Grigory
@pomadchin

@metasim we didnt resolve that; imagine the case that you have multiple RasterSources
you want to figure out how to tile them to some layout in some projection…

it would mean that you probably need to collect metadata in some CRS or reprojet it into some CRS to have some idea of the entire rasters list extent

ahhhh youre talking about persisting
and not about reading
hmmm can you describe your usecase a lil bit more?
Im jsut flygin in a different context at the moment
Simeon H.K. Fitch
@metasim
yeh, sorry. Say you want to write out a bunch of DL chips as geotiffs, organized via some arbitrary partitioning scheme. Maybe some json sidecars for metadata (or not).
BTW, progress is being made: https://circleci.com/gh/s22s/rasterframes/1757

@pomadchin

yeh, sorry. Say you want to write out a bunch of DL chips as geotiffs, organized via some arbitrary partitioning scheme. Maybe some json sidecars for metadata (or not).

Another way of looking at it (from a RasterFrames perspective) is how might I write out unstructured tiles in a form that's not, say Parquet or ORC... some form that could be slurped up by other tooing.

Not really a hard lift... just wondering if you'd gone down that path before.
Nathan Banek
@natonic77

I'm trying to render a GeoTIFF as PNG, but don't know a priori if it is single or multi band. Getting an assertion failure on color when I attempt renderPng if I don't do the following:

val gt = GeoTiffReader.readMultiband("my.tif")
gt.bandCount match {
    case 1 => gt.tile.band(0).renderPng.write("my.png")
    case i if i > 1 => gt.tile.renderPng.write("my.png")
}

I'm a GT newbie, so I'm sure overlooking something obvious, but is there not a single mechanism through the API to render an n-band GeoTIFF?

Grigory
@pomadchin
hey @natonic77 nope, because for multibands and singlebands the way you generate PNGs is different
but if you have a nice API suggestion feel free to throw ideas!
Nathan Banek
@natonic77
thanks for the quick reply @pomadchin. I can work with it, it's just awkward, but I think it may have more to do with me needing to work with the library differently. For my purposes, I probably need to get a little smarter on combining bands since I'm ultimately going to want to render a single layer image.
Grigory
@pomadchin
do you mean a singleband image?
The reason why the render function behaves differently is because renderToPng converts a multibandtile into a single banded png => it means that it needs to combine for instance 3 bands somehow to build a valid png; usually you want to visualize a 3 bands raster (rgb)
In case you have a single band raster it means that you already combined them into a valid band so you can render it
Nathan Banek
@natonic77
yes - I meant singleband, and yes combining into a single band for rendering is the goal. I sometimes receive single band tiffs (grayscale) and other times get RGB multi band tiffs. I was actually trying to use renderJpg to do this originally, but even when calling that from the multi band object it seemed to give me a strangely colored (maybe just blue and red without green) raster, so I had to fall back to renderPng which was giving an accurately colored output (albeit a much larger file). I hadn't gotten far enough with using renderJpg to see that the api mechanism was different.
tosen1990
@tosen1990
Yeah,it has a little bit change.
Grigory
@pomadchin
hey @rexuexiangbei you can use github markdown to wrap your code. Can you throw here exceptions you have?
tosen1990
@tosen1990
@pomadchin Grigory. No worries. I'm telling him in private.
Grigory
@pomadchin
you can talk here as well; thanks!
tosen1990
@tosen1990
x:D
Nathan Banek
@natonic77
Following up on my renderJpg issue, I finally figured out how to get a JPG with proper colors from an RGB multiband geotiff. I wasn't able to use the color() extension method on MultibandTile because it produces RGBA and ImageIO renderJpg wants RGB (and could probably work with ARGB). As before, I'm probably missing something, but I created a colorRgb function to solve my problem. Should I have done something different?
    def colorRgb(t: MultibandTile): Tile = {
      t.convert(IntConstantNoDataCellType).combine(0, 1, 2) { (rBand, gBand, bBand) =>
        val r = if (isData(rBand)) { rBand } else 0
        val g = if (isData(gBand)) { gBand } else 0
        val b = if (isData(bBand)) { bBand } else 0

        if(r + g + b == 0) 0
        else {
          new java.awt.Color(r, g, b).getRGB
        }
      }
    }
Simeon H.K. Fitch
@metasim
:balloon: :balloon: RasterFrames 0.8.4 is Live! :balloon: :balloon:
https://github.com/locationtech/rasterframes/releases/tag/0.8.4
Grigory
@pomadchin
@metasim :tada: :tada: :tada: :tada: GZ! With spark 2.4.4 but still without GT 3
@natonic77 indeed; it is a bit weird that you’re facing this problem, since renderJpg() for multibands uses this method https://github.com/locationtech/geotrellis/blob/master/raster/src/main/scala/geotrellis/raster/render/ColorMethods.scala#L46
which is pretty much the same as yours
renderJpg() works like ~ multiband.color().renderJpg
Nathan Banek
@natonic77
@pomadchin actually, I copied that method to make my version. The difference is that that method produces RGBA rather than ARGB (note that alpha is in the low order bits rather than high order). The Java Color method puts alpha in the high order bits.
Grigory
@pomadchin
@natonic77 aaahhhh gotcha
Grigory
@pomadchin
@natonic77 so we need to fix our to jpeg method I guess?
would you create an issue? I think awt.Color is pretty heavy for this map step though; you can do the same manually to avoid extra objects allocations
Nathan Banek
@natonic77
will do, and yes, I agree - I was going for something tried and true for my experimentation, but just fixing the bitshift operations would do it
Frederic Guiet
@fguiet
Hi @pomadchin
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4b6ed53a82, pid=37772, tid=0x00007f315015f700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x6d0a82]  jni_SetByteArrayRegion+0xc2
#
# Core dump written. Default location: /mnt/data08/yarn/local/usercache/guiet/appcache/application_1571662316912_1404/container_e54_1571662316912_1404_01_000002/core or core.37772
#
# An error report file with more information is saved as:
# /mnt/data08/yarn/local/usercache/guiet/appcache/application_1571662316912_1404/container_e54_1571662316912_1404_01_000002/hs_err_pid37772.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Got this error while trying to load las file using :
val las = spark.read.format("geotrellis.pointcloud.spark.datasource").option("path","hdfs:///user/guiet/Orleans_36_rue_de_la_Borde_Fond_L93.las").load()
File is 1.6g
I need to load las file into Hive
maybe I can use
val testDir = new Path(s"hdfs:////user/guiet/test_geotrellis/USGS_LPC_LA_Barataria_2013_15RYN6548_LAS_2015.las")
val numPoints = HadoopPointCloudRDD(testDir)
But how can I convert RDD to DataFrame
1.6g is not a big las file....I need to load a 50giga las file I am worried
Grigory
@pomadchin
@fguiet yo Im very sorry that you faced this issue, it is a known bug / feature / limitation of the current geotrellis-pointcoud implementation. geotrellis/geotrellis-pointcloud#14
it will require much more time and thinking to allow arbitraty file sizes to work with
as a workaround you can split them into smaller chunks via pdal-pipeline
but at this point I don’t have time to look into it; it is a pretty serious issue - so I would be happy to assist you if you would come up with some really good solution
Frederic Guiet
@fguiet
hummm I see...ok I am gonna split my las file into chunks via pdal split command