spark-submitto download packages from nonstandard locations. Perhaps there's a way to do that with
if you still need GT snapshots than they are available on the maven nexus i.e.: https://oss.sonatype.org/content/repositories/snapshots/org/locationtech/geotrellis/geotrellis-spark_2.12/
Check out the GT README badges: https://github.com/locationtech/geotrellis#geotrellis
Thanks @pomadchin. I tried rasterframes 0.10.1 and the GT errors were resolved. Now there's only one module not found.
com.github.everit-org.json-schema#org.everit.json.schema;1.12.2: not found
I'm simply going through the getting started guide for rasterframes (https://rasterframes.io/getting-started.html) and trying to follow the "using pyspark shell" section, and I get the above error. The getting started guide doesn't appear to work with 0.10.0 and 0.10.1
Hi @jpolchlo, thank you. i'm able to get to the pyspark shell now, but get the following error when i run: spark = spark.withRasterFrames(). From what I've read this looks like a scala version mismatch, but I verified I'm using spark 3.1 which uses scala 2.12
: java.lang.NoSuchMethodError: shapeless.DefaultSymbolicLabelling$.instance(Lshapeless/HList;)Lshapeless/DefaultSymbolicLabelling; at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder(StandardEncoders.scala:68) at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder$(StandardEncoders.scala:68)
set -ex # Install Conda wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh sudo sh Miniconda3-latest-Linux-x86_64.sh -b -p /usr/local/miniconda source ~/.bashrc export PATH=/usr/local/miniconda/bin:$PATH # Install GDAL sudo /usr/local/miniconda/bin/conda config --add channels conda-forge sudo /usr/local/miniconda/bin/conda install -c conda-forge libnetcdf gdal=3.5.0 -y sudo /usr/local/miniconda/bin/pip install pyrasterframes geopandas boto3 s3fs echo "export PATH=/usr/local/miniconda/bin:$PATH" >> ~/.bashrc echo "export LD_LIBRARY_PATH=/usr/local/miniconda/lib/:/usr/local/lib:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib" >> ~/.bashrc echo "export PROJ_LIB=/usr/local/miniconda/share/proj" >> ~/.bashrc echo "export PYSPARK_PYTHON=/usr/local/miniconda/bin/python" >> ~/.bashrc echo "export PYSPARK_DRIVER_PYTHON=/usr/local/miniconda/bin/python" >> ~/.bashrc
But now to hammer you with more questions... I've been using the polygonal summary method in geotrellis over blocks from rasters, and it seems like geotrellis has an optimization where it just rasterizes the parts of the polygon that are within the extent of the raster. Is there a good way to replicate this behavior with rasterframes?
I.e. if I do a big join of a bunch of polygons and raster blocks, and now I want to rasterize the polygons to use as a mask, how do I rasterize so that the zone raster is aligned with just that block? In the example in the documentation, it just rasterizes using the dimensions of the raster, but I'm unclear how this actually aligns correctly with raster block: https://rasterframes.io/zonal-algebra.html