by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 04 18:19

    pomadchin on master

    Fix issue with resource managem… (compare)

  • Jun 04 18:19
    pomadchin closed #3249
  • Jun 04 17:03
    lossyrob commented #3249
  • Jun 04 17:02
    lossyrob review_requested #3249
  • Jun 04 17:00
    lossyrob synchronize #3249
  • Jun 04 14:42
    pomadchin commented #3249
  • Jun 04 14:10
    pomadchin commented #3249
  • Jun 04 03:07
    pomadchin assigned #3249
  • Jun 04 03:07
    pomadchin review_requested #3249
  • Jun 04 02:53
    lossyrob edited #3249
  • Jun 04 02:53
    lossyrob edited #3249
  • Jun 04 02:46
    lossyrob commented #3249
  • Jun 04 02:43
    lossyrob synchronize #3249
  • Jun 04 02:42
    lossyrob commented #3249
  • Jun 04 01:53
    lossyrob synchronize #3249
  • Jun 04 01:35
    lossyrob opened #3249
  • Jun 03 15:00
    pomadchin labeled #3246
  • Jun 02 16:32
    pomadchin edited #3248
  • Jun 02 16:27
    pomadchin labeled #3248
  • Jun 02 16:27
    pomadchin opened #3248
Grigory
@pomadchin
which is pretty much the same as yours
renderJpg() works like ~ multiband.color().renderJpg
Nathan Banek
@natonic77
@pomadchin actually, I copied that method to make my version. The difference is that that method produces RGBA rather than ARGB (note that alpha is in the low order bits rather than high order). The Java Color method puts alpha in the high order bits.
Grigory
@pomadchin
@natonic77 aaahhhh gotcha
Grigory
@pomadchin
@natonic77 so we need to fix our to jpeg method I guess?
would you create an issue? I think awt.Color is pretty heavy for this map step though; you can do the same manually to avoid extra objects allocations
Nathan Banek
@natonic77
will do, and yes, I agree - I was going for something tried and true for my experimentation, but just fixing the bitshift operations would do it
Frederic Guiet
@fguiet
Hi @pomadchin
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4b6ed53a82, pid=37772, tid=0x00007f315015f700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x6d0a82]  jni_SetByteArrayRegion+0xc2
#
# Core dump written. Default location: /mnt/data08/yarn/local/usercache/guiet/appcache/application_1571662316912_1404/container_e54_1571662316912_1404_01_000002/core or core.37772
#
# An error report file with more information is saved as:
# /mnt/data08/yarn/local/usercache/guiet/appcache/application_1571662316912_1404/container_e54_1571662316912_1404_01_000002/hs_err_pid37772.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Got this error while trying to load las file using :
val las = spark.read.format("geotrellis.pointcloud.spark.datasource").option("path","hdfs:///user/guiet/Orleans_36_rue_de_la_Borde_Fond_L93.las").load()
File is 1.6g
I need to load las file into Hive
maybe I can use
val testDir = new Path(s"hdfs:////user/guiet/test_geotrellis/USGS_LPC_LA_Barataria_2013_15RYN6548_LAS_2015.las")
val numPoints = HadoopPointCloudRDD(testDir)
But how can I convert RDD to DataFrame
1.6g is not a big las file....I need to load a 50giga las file I am worried
Grigory
@pomadchin
@fguiet yo Im very sorry that you faced this issue, it is a known bug / feature / limitation of the current geotrellis-pointcoud implementation. geotrellis/geotrellis-pointcloud#14
it will require much more time and thinking to allow arbitraty file sizes to work with
as a workaround you can split them into smaller chunks via pdal-pipeline
but at this point I don’t have time to look into it; it is a pretty serious issue - so I would be happy to assist you if you would come up with some really good solution
Frederic Guiet
@fguiet
hummm I see...ok I am gonna split my las file into chunks via pdal split command
Check Iqumuls library as it seems to be able to load big las file into Hive
worth a look at the implementation at the implementation
Grigory
@pomadchin
Yes, i know about this library. It is completely different and much more restrictive
Frederic Guiet
@fguiet
moreover it depracated and works only with spark 1.6...that a shame
thank to point me this known limition
limitation
Grigory
@pomadchin
well, it is restrctive in the sense of operations and file formats it supports
pointclouds are not only las / laz files
Frederic Guiet
@fguiet
yeah of course
Grigory
@pomadchin
and the implemenataion details are completely different; in our case we’re hitting a problem of trying to allocate a single array for the entire las file
Frederic Guiet
@fguiet
but las file can be very big...so gt-pointcloud must handle this otherwise it will be useless
Grigory
@pomadchin
not really. usually you don’t need all the dimensions to be loaded into memory and not everything at once
Frederic Guiet
@fguiet
single array for the entire las file!!!
Grigory
@pomadchin
it is doable and we would be happy to work on it once it would be required for someone
Frederic Guiet
@fguiet
u usre?
Grigory
@pomadchin
Well yes, Im sure - because I was the one who worked on all the PDAL / jni interfaces and geotrellis-pointcloud itself
Frederic Guiet
@fguiet
yeah but are you sure it is the good implementation
as a las file can contain billion of point
Grigory
@pomadchin
em; I didnt tell that it is a good implementation
it is a naive implementation
Frederic Guiet
@fguiet
:)
anyway, will try to chunk my las file
Grigory
@pomadchin
or you can create a PR and fix the way spark loads pointclouds into memory ;)
Frederic Guiet
@fguiet
so I can load it X,Y,Z values into a big hive table
Grigory
@pomadchin
Also, if you have only x,y,z you can filter the file by dimensions
Frederic Guiet
@fguiet
I will make a PR if I was smarter enough for sure
no the cas thought
case
not