Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 09 12:33

    pomadchin on master

    Bump Scala version and SBT (#34… (compare)

  • Jun 09 12:33
    pomadchin closed #3400
  • Jun 08 20:59
    pomadchin synchronize #3400
  • Jun 08 20:50
    pomadchin synchronize #3400
  • Jun 08 20:49
    pomadchin edited #3400
  • Jun 08 20:48
    pomadchin opened #3400
  • Jun 07 18:47

    pomadchin on npm_and_yarn

    (compare)

  • Jun 07 18:47

    pomadchin on master

    Bump ws from 6.2.1 to 6.2.2 in … (compare)

  • Jun 07 18:47
    pomadchin closed #3399
  • Jun 06 15:21
    dependabot[bot] labeled #3399
  • Jun 06 15:21
    dependabot[bot] opened #3399
  • Jun 06 15:21

    dependabot[bot] on npm_and_yarn

    Bump ws from 6.2.1 to 6.2.2 in … (compare)

  • May 29 13:40

    pomadchin on npm_and_yarn

    (compare)

  • May 29 13:40

    pomadchin on master

    Bump dns-packet from 1.3.1 to 1… (compare)

  • May 29 13:40
    pomadchin closed #3398
  • May 29 03:57
    dependabot[bot] labeled #3398
  • May 29 03:57
    dependabot[bot] opened #3398
  • May 29 03:57

    dependabot[bot] on npm_and_yarn

    Bump dns-packet from 1.3.1 to 1… (compare)

  • May 25 03:43

    pomadchin on master

    Update CHANGELOG.md (compare)

  • May 25 03:42

    pomadchin on master

    Add new shading rules to make G… (compare)

Grigory
@pomadchin
ah I see
I guess that is just a corrupted tiff
you can try to fix it via GDALWarp / translate commands (if GDAL at all can handle that)
what software you use to produce such TIFFS? ENVI?
pikanin
@pikanin
yes, ENVI
Grigory
@pomadchin
(: eh in this case sounds like you can file an ENVI bug report
pikanin
@pikanin
i'll try the cases
Grigory
@pomadchin
:rocket:
pikanin
@pikanin
thank you
Grigory
@pomadchin
you're welcome :D but I guess I could not help much
林海听枫
@stjimreal
Hello, Grigory. No way that I did find out the correct format to connect a HDF file! The format shoud be a full name of SUBDATASETs like this: HDF4_EOS:EOS_GRID:\"(your file path/)MCD12Q1.A2019001.h25v09.006.2020212132651.hdf\":MCD12Q1:LC_Type1 And I posted something on (NETCDF / HDF5 Metadata retrieval issue #70)[geotrellis/gdal-warp-bindings#70].
@pomadchin
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG
Hey, could the awesome gt process the tiff with some pre-process method (like radiation calibration or atmospheric correction or uniform color )?
Grigory
@pomadchin
@DDDDDDDDDDDDLANG hey you can do whatever is possible, but there are no built in functions to do these operations, so you’d have to implement it yourself
We have also gdal support (: just in case you were relying on some GDAL capability
DDDDDDDDDDDDLANG
@DDDDDDDDDDDDLANG
@pomadchin Got it! Thx:)
pikanin
@pikanin
@pomadchin The tags of the big Tiff files clipped by ENVI have problems and the same size image generated by the gdal could be read as normal. Hi, the read apis for the single machine (without machine) read the big tiff file is not easy as the memory limitations, while it could deal with bigger tiff files (> 4g) in the cluster environment with Spark.
Althoug the GT is in 2.x version.
Thank you very much.
Grigory
@pomadchin
@pikanin got it, so that's envi issue
Max
@Max-AR
Hi! I am loading a geotiff via RasterSource and I am getting back an Extent that has one of the points set as 0.0. The geotiff looks normal in QGIS and reports all the extents correctly. Does anyone know what could be causing this problem? My code is very simple;
val rasterSource = RasterSource(path)
println(rastersource.extent)
Grigory
@pomadchin

hey @Max-AR nope, what GT version do you use? Can you send here gdalinfo (extent part at least) of the image? Could you send the result returned by the RasterSource?

Do you use GDALRasterSource or GeoTiffRasterSource?

pikanin
@pikanin
@pomadchin got it, thank you!!!
JeroenVerstraelen
@JeroenVerstraelen
Hello, has anyone experimented with integrating Apache Arrow into geopyspark to ensure that dataframes don't need to be copied over and serialized every time they move between Python and Scala processes? I read through some old messages in this chat but it seems like the Arrow discussion died down in 2018.
Grigory
@pomadchin
hey @JeroenVerstraelen we don’t have geopyspark in the active development.
however, you can rely on dataframes and datasets API (see the RasterFrames project) they have (due to the fact of having a dataset / dataframe api) a nice py support
Apache Arrow is supported by Spark natively (though the support is only initial);
talking about geopyspark I also dont think there is much that can be improved; but you are very much weclome to make your opensource contributions!
just for the context: both in spark now and in geo py spark you can at max to try to use Arrow as an alternative serialzier
mem copy would still happen, but due to the fact that it works though the py4j socket, you can’t avoid memory copy
Jeroen Dries
@jdries
Hi @pomadchin , I'm working with Jeroen on this, would it be feasible to avoid the py4j socket issue by allocating the tile memory off heap, and simply passing the pointer?
On geopyspark: we're somewhat diminishing our use of that, and working with pure geotrellis for most of our things. The python connection is mostly there to allow our users to work with Python scripts as well.
Grigory
@pomadchin
@jdries hey, nope, that’s how spark executors invoke py / scala code
but even in theory that is kind of not straighfotwrad, i haven’t seen apache arrow usage as a memory mapping tool; at most the zero-copy (but still a copy) passing blobs between memories
even if it is smth like jep via JNI it is still a memory copy
but in theory memory mapping is possible but should be supported on the Spark side
Grigory
@pomadchin
@jdries do you have any up to date examples of the apache arrow mem mapping between native mem and java?
I haven’t looked into apache arrow for a while already
Jeroen Dries
@jdries
No, also didn't check it out fully, @JeroenVerstraelen is doing that.
What I had in mind does look a lot like what is described here:
https://github.com/ninia/jep/blob/master/release_notes/3.7-notes.rst#shared-ndarrays
Which seems to claim 'there is no need to copy the memory'.
Grigory
@pomadchin
@jdries there is always a need, that is not a pointer; DirectBuffers allocate bytes in C memory; but when you ‘read it’ it performs a copy into the JVM memory
that’s how JVM works
so what is the possible way: allocate buffer, but data into it from py, and ‘read’ it from Java
to perform java ops on tiles
that is how JNI works
by the end of the day you’d have to ‘fill in’ the new direct buffer (or the old), that would be the copy in another direction
I agree that would be faster than py4j and it is more efficient
I had a good experience with using jep to interact with the py code
what can be possible (via my experience) is to have some apache arrow structure and to pass it around via arrow; in this case we’re in fact copying just a chunk of memory that is read and written the same way both in c and jvm
Grigory
@pomadchin
what is more interesting is https://github.com/NVIDIA/spark-rapids but I guess they also use Arrow as a serialization format
Jeroen Dries
@jdries
Thanks a lot, we're going to check out some of the options a bit further!
Grigory
@pomadchin
@jdries let me know about the options you’d decide; I’m pretty much interested in the collab so dont hesistate to drop messages into this chat :wave: