^ try to add https://jitpack.io as a resolver => it should work than
@pomadchin thanks for the response. Forgive me for the noob questions :). I'm a python developer and have very little understanding of what's happening behind the scenes with the RF zip file. What would I need to add to the command from the "usying pyspark shell" section to add a resolver?
Hi @jpolchlo, thank you. i'm able to get to the pyspark shell now, but get the following error when i run: spark = spark.withRasterFrames(). From what I've read this looks like a scala version mismatch, but I verified I'm using spark 3.1 which uses scala 2.12
: java.lang.NoSuchMethodError: shapeless.DefaultSymbolicLabelling$.instance(Lshapeless/HList;)Lshapeless/DefaultSymbolicLabelling; at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder(StandardEncoders.scala:68) at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder$(StandardEncoders.scala:68)
@kembles5 Yes, that looks like some kind of version mismatch, but the origin of those can be hard to track down. Are you setting up a very vanilla, minimal environment here, or are there other dependencies getting thrown in here? I haven't worked too much with pyrasterframes, so I'm possibly not extremely useful here. I wonder if @metasim has any advice?
Thanks @jpolchlo. Looks like the latest version of spark is using shapeless 2.3.7 (per pom.xml from master branch) and the version that 0.10.1 depends on is 2.12. Is there another version of the rasterframes zip file (maybe a shaded version) that I should be using with spark 3. I'm using pyrasterframes_2.12-0.10.0-python.zip
I think we need to move to spark 3.2.x to get rid of this error and be fully compatible with everything
@pomadchin I can't upgrade shapeless in the spark classpath and I can't move to spark 3.2. That leaves using the assembly jar with shaded deps. Where can I get this jar, i've searched high and low. I also tried to build the project but run into the following error(macos, openjdk18):
java.lang.ClassCastException: class java.lang.UnsupportedOperationException cannot be cast to class xsbti.FullReload (java.lang.UnsupportedOperationException is in module java.base of loader 'bootstrap'; xsbti.FullReload is in unnamed module of loader 'app')
sbt script version: 1.6.2
That's my personal laptop, but at work and where I'm using spark, I'm on rhel7
I would recommend you to follow RF quick start instructions
@kembles5 check this out https://github.com/locationtech/rasterframes/issues/581#issuecomment-1086542343
a good question how to deploy it on a cluster with no k8s, that’s a q to @metasim, but I’m pretty sure it is doable via conda
@pomadchin Thanks for the info. The getting started guide is where I'm having issues. Specifically, the "Using pyspark shell" section. I tried following the guide using 0.10.0 and 0.10.1 on spark 3.1. Both initially failed with "modules not found", adding --repositories https://jitpack.io had no effect, although 0.10.1 eventually started working despite no changes, the 0.10.0 continues to have issues with finding the geotrellis module. For 0.10.1 i then ran into the shapeless conflict.
Hey @jterry64 I think it can be some version of https://github.com/pomadchin/vlm-performance/blob/feature/gt-3.x/emr/bootstrap.sh but with an extra RasterFrames deps
Woo I did finally get it to work! This ended up working as a bootstrap scripts:
set -ex
# Install Conda
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sudo sh Miniconda3-latest-Linux-x86_64.sh -b -p /usr/local/miniconda
source ~/.bashrc
export PATH=/usr/local/miniconda/bin:$PATH
# Install GDAL
sudo /usr/local/miniconda/bin/conda config --add channels conda-forge
sudo /usr/local/miniconda/bin/conda install -c conda-forge libnetcdf gdal=3.5.0 -y
sudo /usr/local/miniconda/bin/pip install pyrasterframes geopandas boto3 s3fs
echo "export PATH=/usr/local/miniconda/bin:$PATH" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/local/miniconda/lib/:/usr/local/lib:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib" >> ~/.bashrc
echo "export PROJ_LIB=/usr/local/miniconda/share/proj" >> ~/.bashrc
echo "export PYSPARK_PYTHON=/usr/local/miniconda/bin/python" >> ~/.bashrc
echo "export PYSPARK_DRIVER_PYTHON=/usr/local/miniconda/bin/python" >> ~/.bashrcBut now to hammer you with more questions... I've been using the polygonal summary method in geotrellis over blocks from rasters, and it seems like geotrellis has an optimization where it just rasterizes the parts of the polygon that are within the extent of the raster. Is there a good way to replicate this behavior with rasterframes?
I.e. if I do a big join of a bunch of polygons and raster blocks, and now I want to rasterize the polygons to use as a mask, how do I rasterize so that the zone raster is aligned with just that block? In the example in the documentation, it just rasterizes using the dimensions of the raster, but I'm unclear how this actually aligns correctly with raster block: https://rasterframes.io/zonal-algebra.html
/export/server/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--jars /export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/jars/pyrasterframes-assembly-0.10.1.jar \
/tmp/pycharm_project_255/raster_slope.py
--master yarn \
--num-executors 6 \
--jars /export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/jars/pyrasterframes-assembly-0.10.1.jar \
/tmp/pycharm_project_255/raster_slope.py
Using spark-submit will cause an error as follows:
Traceback (most recent call last):
File "/tmp/pycharm_project_255/raster_slope.py", line 9, in <module>
spark = (SparkSession.builder
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/init.py", line 44, in _rf_init
spark_session.rasterframes = RFContext(spark_session)
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/rf_context.py", line 45, in init
self._jrfctx = self._jvm.org.locationtech.rasterframes.py.PyRFContext(jsess)
File "/export/server/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in call
File "/export/server/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
File "/export/server/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.locationtech.rasterframes.py.PyRFContext.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.objects.Invoke$.apply$default$5()Z
at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:154)
at scala.collection.immutable.List.map(List.scala:293)
at frameless.RecordEncoder.toCatalyst(RecordEncoder.scala:153)
at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
at org.locationtech.rasterframes.encoders.TypedEncoders.typedExpressionEncoder(TypedEncoders.scala:22)
at org.locationtech.rasterframes.encoders.TypedEncoders.typedExpressionEncoder$(TypedEncoders.scala:22)
at org.locationtech.rasterframes.package$.typedExpressionEncoder(package.scala:39)
at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder(StandardEncoders.scala:68)
at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder$(StandardEncoders.scala:68)
at org.locationtech.rasterframes.package$.spatialKeyEncoder$lzycompute(package.scala:39)
at org.locationtech.rasterframes.package$.spatialKeyEncoder(package.scala:39)
at org.locationtech.rasterframes.StandardColumns.$init$(StandardColumns.scala:42)
at org.locationtech.rasterframes.package$.<init>(package.scala:39)
at org.locationtech.rasterframes.package$.<clinit>(package.scala)
at org.locationtech.rasterframes.py.PyRFContext.<init>(PyRFContext.scala:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
File "/tmp/pycharm_project_255/raster_slope.py", line 9, in <module>
spark = (SparkSession.builder
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/init.py", line 44, in _rf_init
spark_session.rasterframes = RFContext(spark_session)
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyrasterframes/rf_context.py", line 45, in init
self._jrfctx = self._jvm.org.locationtech.rasterframes.py.PyRFContext(jsess)
File "/export/server/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in call
File "/export/server/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
File "/export/server/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.locationtech.rasterframes.py.PyRFContext.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.objects.Invoke$.apply$default$5()Z
at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:154)
at scala.collection.immutable.List.map(List.scala:293)
at frameless.RecordEncoder.toCatalyst(RecordEncoder.scala:153)
at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
at org.locationtech.rasterframes.encoders.TypedEncoders.typedExpressionEncoder(TypedEncoders.scala:22)
at org.locationtech.rasterframes.encoders.TypedEncoders.typedExpressionEncoder$(TypedEncoders.scala:22)
at org.locationtech.rasterframes.package$.typedExpressionEncoder(package.scala:39)
at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder(StandardEncoders.scala:68)
at org.locationtech.rasterframes.encoders.StandardEncoders.spatialKeyEncoder$(StandardEncoders.scala:68)
at org.locationtech.rasterframes.package$.spatialKeyEncoder$lzycompute(package.scala:39)
at org.locationtech.rasterframes.package$.spatialKeyEncoder(package.scala:39)
at org.locationtech.rasterframes.StandardColumns.$init$(StandardColumns.scala:42)
at org.locationtech.rasterframes.package$.<init>(package.scala:39)
at org.locationtech.rasterframes.package$.<clinit>(package.scala)
at org.locationtech.rasterframes.py.PyRFContext.<init>(PyRFContext.scala:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
Run the project directly won't cause any error. But I want to set num-executors
Trying to debug a GDAL reading problem that results in following exception (on EMR with GDAL 3.1.2 installed)
Caused by: java.lang.UnsupportedOperationException: Reading 'gdal://vsis3/bucket/some.tif not supported
at org.locationtech.rasterframes.ref.RFRasterSource$.$anonfun$apply$1(RFRasterSource.scala:119)
at scala.compat.java8.functionConverterImpls.AsJavaFunction.apply(FunctionConverters.scala:262)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)rasterframes/RFRasterSource.scala at develop · locationtech/rasterframes · GitHub
In spark-shell on master with the job jar I’m able to reproduce but not explain:
Raster Reads:
val url = "gdal://vsis3/bucket/some.tif”
scala> val rs = RFRasterSource(new java.net.URI(url))
rs: org.locationtech.rasterframes.ref.RFRasterSource = GDALRasterSource(gdal://vsis3/...)
scala> rs.read(GridBounds(0,0,10,10), List(0))
res1: geotrellis.raster.Raster[geotrellis.raster.MultibandTile] = Raster(ArrayMultibandTile(11,11,1,float32ud-3.4028234663852886E38),Extent(4031670.65908466, 3215321.1233700267, 4031725.65908466, 3215376.1233700267))
scala> RFRasterSource.IsGDAL.unapply(new java.net.URI(url))
res2: Boolean = true
scala> spark.read.raster.from(url).load().show()
22/06/09 19:26:23 WARN TaskSetManager: Lost task 986.0 in stage 1.0 (TID 988) (ip-172-31-19-30.eu-west-1.compute.internal executor 7): java.lang.IllegalArgumentException: Error fetching data for one of:
at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:83)
at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$3(GenerateExec.scala:95)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at scala.collection.Iterator$ConcatIterator.hasNext(Iterator.scala:222)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:275)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:400)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.UnsupportedOperationException: Reading 'gdal://vsis3/...
8 replies
@metasim Related to the question above, I’ve been noticing that GeoTiff reader is just kind of really really bad compared to GDAL when reading tiles. Much worse than I would have expected it to be based on previous expirience with it. I wonder if that lines up with what you’ve seen and if you, per-chance, have a theory as to what is going on. It’d be really nice to be able to handle COGs without GDAL. Especially now that we have EMR Serverless as an option.