geotrellis/geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.

Grigory

@pomadchin

I guess that is just a corrupted tiff

you can try to fix it via GDALWarp / translate commands (if GDAL at all can handle that)

what software you use to produce such TIFFS? ENVI?

pikanin

@pikanin

yes, ENVI

Grigory

@pomadchin

(: eh in this case sounds like you can file an ENVI bug report

pikanin

@pikanin

i'll try the cases

Grigory

@pomadchin

:rocket:

pikanin

@pikanin

thank you

Grigory

@pomadchin

you're welcome :D but I guess I could not help much

林海听枫

@stjimreal

Hello, Grigory. No way that I did find out the correct format to connect a HDF file! The format shoud be a full name of SUBDATASETs like this: HDF4_EOS:EOS_GRID:\"(your file path/)MCD12Q1.A2019001.h25v09.006.2020212132651.hdf\":MCD12Q1:LC_Type1 And I posted something on (NETCDF / HDF5 Metadata retrieval issue #70)[geotrellis/gdal-warp-bindings#70].

@pomadchin

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

Hey, could the awesome gt process the tiff with some pre-process method (like radiation calibration or atmospheric correction or uniform color )?

Grigory

@pomadchin

@DDDDDDDDDDDDLANG hey you can do whatever is possible, but there are no built in functions to do these operations, so you’d have to implement it yourself

We have also gdal support (: just in case you were relying on some GDAL capability

DDDDDDDDDDDDLANG

@DDDDDDDDDDDDLANG

@pomadchin Got it! Thx:)

pikanin

@pikanin

@pomadchin The tags of the big Tiff files clipped by ENVI have problems and the same size image generated by the gdal could be read as normal. Hi, the read apis for the single machine (without machine) read the big tiff file is not easy as the memory limitations, while it could deal with bigger tiff files (> 4g) in the cluster environment with Spark.

Althoug the GT is in 2.x version.

Thank you very much.

Grigory

@pomadchin

@pikanin got it, so that's envi issue

well we have an example of raster sources usage to read files into RDDs https://github.com/pomadchin/vlm-performance/blob/feature/gt-3.x/src/main/scala/geotrellis/contrib/performance/IngestRasterSource.scala#L52-L59

Max

@Max-AR

Hi! I am loading a geotiff via RasterSource and I am getting back an Extent that has one of the points set as 0.0. The geotiff looks normal in QGIS and reports all the extents correctly. Does anyone know what could be causing this problem? My code is very simple;

val rasterSource = RasterSource(path)
println(rastersource.extent)

Grigory

@pomadchin

hey @Max-AR nope, what GT version do you use? Can you send here gdalinfo (extent part at least) of the image? Could you send the result returned by the RasterSource?

Do you use GDALRasterSource or GeoTiffRasterSource?

pikanin

@pikanin

@pomadchin got it, thank you!!!

JeroenVerstraelen

@JeroenVerstraelen

Hello, has anyone experimented with integrating Apache Arrow into geopyspark to ensure that dataframes don't need to be copied over and serialized every time they move between Python and Scala processes? I read through some old messages in this chat but it seems like the Arrow discussion died down in 2018.

Grigory

@pomadchin

hey @JeroenVerstraelen we don’t have geopyspark in the active development.
however, you can rely on dataframes and datasets API (see the RasterFrames project) they have (due to the fact of having a dataset / dataframe api) a nice py support

Apache Arrow is supported by Spark natively (though the support is only initial);

talking about geopyspark I also dont think there is much that can be improved; but you are very much weclome to make your opensource contributions!

just for the context: both in spark now and in geo py spark you can at max to try to use Arrow as an alternative serialzier

mem copy would still happen, but due to the fact that it works though the py4j socket, you can’t avoid memory copy

Jeroen Dries

@jdries

Hi @pomadchin , I'm working with Jeroen on this, would it be feasible to avoid the py4j socket issue by allocating the tile memory off heap, and simply passing the pointer?
On geopyspark: we're somewhat diminishing our use of that, and working with pure geotrellis for most of our things. The python connection is mostly there to allow our users to work with Python scripts as well.

Grigory

@pomadchin

@jdries hey, nope, that’s how spark executors invoke py / scala code

but even in theory that is kind of not straighfotwrad, i haven’t seen apache arrow usage as a memory mapping tool; at most the zero-copy (but still a copy) passing blobs between memories

even if it is smth like jep via JNI it is still a memory copy

but in theory memory mapping is possible but should be supported on the Spark side

Grigory

@pomadchin

@jdries do you have any up to date examples of the apache arrow mem mapping between native mem and java?

I haven’t looked into apache arrow for a while already

Jeroen Dries

@jdries

No, also didn't check it out fully, @JeroenVerstraelen is doing that.
What I had in mind does look a lot like what is described here:
https://github.com/ninia/jep/blob/master/release_notes/3.7-notes.rst#shared-ndarrays
Which seems to claim 'there is no need to copy the memory'.

Grigory

@pomadchin

@jdries there is always a need, that is not a pointer; DirectBuffers allocate bytes in C memory; but when you ‘read it’ it performs a copy into the JVM memory

that’s how JVM works

so what is the possible way: allocate buffer, but data into it from py, and ‘read’ it from Java

to perform java ops on tiles

that is how JNI works

by the end of the day you’d have to ‘fill in’ the new direct buffer (or the old), that would be the copy in another direction

I agree that would be faster than py4j and it is more efficient

I had a good experience with using jep to interact with the py code

what can be possible (via my experience) is to have some apache arrow structure and to pass it around via arrow; in this case we’re in fact copying just a chunk of memory that is read and written the same way both in c and jvm

Grigory

@pomadchin

what is more interesting is https://github.com/NVIDIA/spark-rapids but I guess they also use Arrow as a serialization format

Jeroen Dries

@jdries

Thanks a lot, we're going to check out some of the options a bit further!

Grigory

@pomadchin

@jdries let me know about the options you’d decide; I’m pretty much interested in the collab so dont hesistate to drop messages into this chat :wave:

Grigory

@pomadchin

Probably not everyone follows proj4j releases (they are pretty rare!) but we have a new 1.1.3 release that fixes omerc projection :rocket: https://github.com/locationtech/proj4j/releases/tag/v1.1.3

Where communities thrive

People

Repo info

Activity