GeoTrellis is a geographic data processing engine for high performance applications.
pomadchin on master
Cleanup build files (#3386) (compare)
pomadchin on npm_and_yarn
pomadchin on master
Bump ssri from 6.0.1 to 6.0.2 i… (compare)
dependabot[bot] on npm_and_yarn
Bump ssri from 6.0.1 to 6.0.2 i… (compare)
pomadchin on master
Update CHANGELOG.md (compare)
pomadchin on master
Cleanup HBase dependencies (#33… (compare)
so blocking-thread-pool.threads seems to help transfer stuff from S3 faster
if i understood it correctly it is used inside BlockingThreadPool which is used in a variety of S3Layer readers and writers
so simple example
having an executor with 4 cores and setting geotrellis.blocking-thread-pool.threads to 20 means
reading/writing to/from S3 is done with 20 threads in parallel but mapping steps are done with 3 parallel threads
that's why i see the 16 geotrellis threads as waiting in a map step
did i get it right?
and the reason why it makes sense in your case to allocate more threads is because you have lots of unused CPUs, you have a nice bandwidth, but the internet connection is pretty limited
so you can slowly fetch more data in parallel
threadpool is allocated in a such a way that it is a single thread pool per executor, so it’s not that expensive and you don’t need to worry about the amount of threadpools
that is not a regular practice in spark, usually parallelism is achieved more traditionally
by having smaller partitions and by having more thin executors
Hey guys, some GT news here! We successfully dropped Scala 2.11 support and established Scala 2.13 cross compilation :tada:
GT 2.13 version depends on the Spark 3.2.0 SNAPSHOT so be careful using 2.13 sparky deps. Check out new artifacts in the Eclipse snapshots repo here https://repo.eclipse.org/content/repositories/geotrellis-snapshots/org/locationtech/geotrellis/
We also upgraded the current master up to Spark 3.0.1 (to be compatible with EMR) and upgraded our Hadoop dependency (to match EMR version) up to 3;
These changes allowed us to add GT JDK 11 builds support (with the help of this tiny PR locationtech/geotrellis#3383)
@FireByTrial yea I don’t think you need any 3d part ydeps; the idea is to read file via geotools, geotools would represent it as a SimpleFeature, and you can extract geometry from the geotools simple feature (via https://github.com/geotools/geotools/blob/main/modules/library/opengis/src/main/java/org/opengis/feature/simple/SimpleFeature.java#L273) and this geometry is a JTS geometry
^ all steps above are implemented and have convenient methods in the geotrellis-geotools package
isNoDataTile
might do what I am after
geotrellis.spark.clip.ClipToGrid
(geotrellis v 2.3.1 still) and I get a bunch of org.locationtech.jts.geom.TopologyException
s on Polygon data where I don't own the source... side location conflicts and non-noded linestring intersections... from general googling, it seems like it's often rounding problems w/ high precision xy data... is there some generalized approach to handling this kind of thing, or should I get busy writing data cleaning methods to modify the data?
.buffer(0)
)