These are chat archives for thunder-project/thunder
To answer my own questions above:
I believe loadImages() does not parallelize because images cannot be parallelized in Spark. Is this correct? Once I load the images as binaries, things seemed to parallelize correctly. @freeman-lab Could you confirm that this is the case about loading images?
To solve the Java Heap Space OutOfMemoryError, I added --driver-memory #G to thunder-submit, where # was the amount of memory on the node I reserved through submission on the cluster. This is without using export _JAVA_OPTIONS="-Xms512m -Xmx4g" to my ~/.bash_profile. Would there be an additional reason to include the additional _JAVA_OPTIONS environment variable?