by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Dave Johnson
@djohnson729
I actually just changed that page this morning based on your feedback. So you'd want to use $MRGEO_COMMON_HOME/bin in your path of course.
Dave S.
@dsrebro
image.png
This message was deleted
image.png
Dave S.
@dsrebro
@djohnson729 I ran an ingest on a large dataset on my desktop, however no file was created in HDFS. DFS is running fine with two slaves. But to me it looks like MrGeo doesn't have anything to ingest, there is about 4000 tif files(22gb) in the directory on the desktop. see screenshots above. thanks.
Dave Johnson
@djohnson729
If you run "hadoop fs -ls /mrgeo/raw-images" does it list train-tif.tif? It doesn't seem to be finding the input image.
Dave S.
@dsrebro
No the tif folder didn't copy to /mrgeo/raw-images
Dave Johnson
@djohnson729
You have a couple of choices for ingest. You can either have your source files in your local file system or in HDFS. If you wish to ingest them from your local filesystem, use "mrgeo ingest -o my-image file:///home/myself/source/my-images" for example. You can point it to a directory containing multiple images (in which case it will ingest all of the images together into a single MrGeo image). You can also specify a single image file, in which case it ingests that single file into a MrGeo image. If you wish to ingest your files from HDFS, then you have to put your source images into HDFS first using "hadoop fs -put".
I also noticed that your -o argument looked like a local file system path. Was that intentional?
You'll want your output (MrGeo image) stored in HDFS (or S3 if that is of interest to you)
leiserhartbeck
@leiserhartbeck
Hello everyone. I am new to MrGeo. Where do I find Legion GPU code in MrGeo and How do I compile it? In addition how do I perform terrain analysis as shown in video? Any help is appreceated.
Dave Johnson
@djohnson729
@leiserhartbeck Legion is not an open source project. What sort of terrain analytics are you interested in? MrGeo has a lot of capability available through map algebra that would probably prove useful to you.
leiserhartbeck
@leiserhartbeck
@djohnson729 Hi Dave. Thanks for the info. I wish to analyse terrain data for some simple calculations like Line of sight to and from a point. Also how do I do terrain analytics for water flowing off a stream ?
Dave Johnson
@djohnson729
@leiserhartbeck Do you need to do large scale processing? Or can your data fit into memory on a single machine?
leiserhartbeck
@leiserhartbeck
@djohnson729 yes it can fit on a single machine
Dave Johnson
@djohnson729
@leiserhartbeck In that case I would suggest using something other than MrGeo then. MrGeo is targeted at large-scale image processing (e.g. worldwide or very high-res imagery over a large area). For your problem, it would introduce unnecessary complexity. Have you taken a look at the QGIS hydrological tools to see if it has what you need at least for the water flow problem?
Josée-Anne Langlois
@jalanglois1_twitter
Hello. I'm trying to ingest a tif image in mrgeo with the ingest command, but I have an Out of Memory issue. The image is a dtm of 19Mo. I tried to increase spark.executor.memory and spark.driver.memory parameters in the spark-default.conf file without any success. I tried to load directly from the file, as well as to load from hdfs with same result. Thanks
Dave Johnson
@djohnson729
@jalanglois1_twitter What is the size of your largest individual source image file? Are you ingesting a directory of files or just one? If ingesting a directory, how many files? As background info, the ingest uses GDAL to open the image and reproject it to WGS84. What is likely happening is that Spark is consuming enough memory on the machine that there is not enough for GDAL. Ingest needs enough memory to store the image twice - once for the source image itself and once for the reprojected image. There are a couple of ways to handle this. First, if the machine has enough memory available to support the Spark executor and storing the image twice in memory, then you should be able to set spark.yarn.executor.memoryOverhead to an appropriate value to make sure Spark leaves enough memory for GDAL. I have not personally tried that, but in theory it should work. The other approach is to to tile your source image using gdal_translate to shrink the size of each individual image so that it reduces the memory burden during ingest. Another nice result of this approach is that it will distribute out the ingest processing. If you choose the gdal_translate approach and you have a lot of source images that need to be tiled, you can use Spark to speed up the process in a Databricks or EMR cluster. I can provide some more details on that if needed.
Josée-Anne Langlois
@jalanglois1_twitter
Thanks for your answer! I have an individual .tif file of 73 Mo (10000X10000 pixels).
Josée-Anne Langlois
@jalanglois1_twitter
@djohnson729 I finally managed to ingest my file by increasing the HADOOP_CLIENT_OPTS .

I'm now trying to build the mrgeo-geoserver plugin, but I have this error :
[ERROR] Failed to execute goal on project mrgeo-geoserver: Could not resolve dependencies for project org.mrgeo:mrgeo-geoserver:jar:1.1: Failed to collect dependencies for [log4j:log4j:jar:1.2.14 (compile), org.mrgeo:mrgeo-core:jar:1.2.0-apache2.7.3-SNAPSHOT (compile), org.geoserver:gs-main:jar:2.9.0 (provided), org.geotools:gt-main:jar:14.0 (provided), org.geotools:gt-data:jar:14.0 (provided), org.geotools:gt-coverage:jar:14.0 (provided), javax.media:jai_core:jar:1.1.3 (provided), javax.media:jai_imageio:jar:1.1 (provided), org.apache.httpcomponents:httpclient:jar:4.5 (compile), org.apache.hadoop:hadoop-common:jar:2.7.2 (compile), org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.2 (compile), org.apache.hadoop:hadoop-hdfs:jar:2.7.2 (compile)]: Failed to read artifact descriptor for org.geoserver:gs-main:jar:2.9.0: Could not transfer artifact org.geoserver:gs-main:pom:2.9.0 from/to boundless (http://repo.boundlessgeo.com/main): peer not authenticated -> [Help 1]

Any idea on what would be the problem? Thanks!

Josée-Anne Langlois
@jalanglois1_twitter
Ok, I solved my problem by using the command sudo update-ca-certificates -f and adding -Dmaven.wagon.http.ssl.insecure=true
-Dmaven.wagon.http.ssl.allowall=true to the mvn clean package command. But now I can't reach the mrgeo maven repository. The link provided in the pom file is not working. http://mrgeo-maven.s3-website-us-east-1.amazonaws.com/. Is it normal?
Dave Johnson
@djohnson729
@jalanglois1_twitter Hmm, it works for me. What is the error you are getting? Which version of maven?
Josée-Anne Langlois
@jalanglois1_twitter
I'm not exactly sure why, but it works today. I finally compiled the plugin using the parameter -Dmrgeo.version=1.2.0-SNAPSHOT.
After installing the plugin, am i suppose to see a new Raster Data Sources type in geoserver ? Is there documentation on how to create WMS based on MrGeo dataset in geoserver?
Dave Johnson
@djohnson729
That plugin exposes MrGeo images as coverages to Geoserver. So what you will see is when you connect to the Geoserver via WMS, the MrGeo that are present should show up in the GetCapabilities request, and of course you make GetMap requests against those layers. Be sure that the configuration described in the "Installing" section at https://github.com/ngageoint/mrgeo-geoserver-plugin is set up for your environment.
Dave Johnson
@djohnson729
I don't know if Geoserver is a requirement for you or not, but another option is to use MrGeo's built-in web server which provides access to imagery through WMS and WCS. It's also really easy to run. After building the distribution, look in distribution/distribution-tgz/target subdir, and you'll see the tar.gz file. Install that as described at https://github.com/ngageoint/mrgeo/wiki/Installation. Once that is done and your environment and mrgeo.conf are set up properly, you can runthe web server as described at https://github.com/ngageoint/mrgeo/wiki/Running-web-server. I do this all the time, and then you can connect to either the WMS or WCS using the URL the web server prints to the console with "mrgeo/wms" or "mrgeo/wcws" appended. By default it is http://localhost:8080/mrgeo/wms.
Dave Johnson
@djohnson729
Oh and if you want to use the MrGeo web server, be sure to set -Pbuild-ogc and -Pstandalone-webserver on your build command line. And another overall build note is to make sure you set the gdal.version property in maven to match the version of GDAL you have installed. If your images are stored in S3, also include the enable-s3a profile in the build.
Josée-Anne Langlois
@jalanglois1_twitter
Thanks for your help! The MrGeo's built-in web server is great and was easy to put in place!
I have another question.. As written in the documentation, the ingest reproject the data in WGS84. Do you plan to support other reference system?
Dave Johnson
@djohnson729
No, that's for storage and processing. We tile the images based on WGS84 internally. However, you can use the WMS to request the image back in a different projection.
You'd just include the CRS/SRS request parameter in your URL
Josée-Anne Langlois
@jalanglois1_twitter
Ok, but we have some concern about error caused by the reprojection process in WGS84. We work with NAD83 UTM projections. We will have to do some test to verify if the error is significant. Would it be a lot of work to implement support for another projection for storage and processing?
Dave Johnson
@djohnson729
Basically, MrGeo requires ingesting the images into WGS-84 for our tiling approach. See https://github.com/ngageoint/mrgeo/wiki/MrsPyramid-Format for more information. What this allows is that when you run map algebra that involves multiple images (e.g. image1 + image2 for a simple example), the tiles of the two images perfectly overlap and we can add corresponding pixel values together.
Josée-Anne Langlois
@jalanglois1_twitter
Ok, I understand.
Another question... is there a way to resample an image to a larger pixel size in mrgeo?
Dave Johnson
@djohnson729
You can ingest to a specific zoom level with the -z argument. Is that what you're looking for?
Dave Johnson
@djohnson729
The other option is that you can ingest the image at its native zoom level (which is does by default), and then in map algebra, you can use the "zoom" function to process a specific zoom level. This requirs of course that you build the pyramid for the image after ingesting.
Josée-Anne Langlois
@jalanglois1_twitter
Ok, then I would have to find a zoom level which correspond the best to the resolution we are looking for in our projected system for our data coverage which will potentially be the entire Canada....
Dave Johnson
@djohnson729
Josée-Anne Langlois
@jalanglois1_twitter
Ok thanks!
Dave Johnson
@djohnson729
ANy time!
Josée-Anne Langlois
@jalanglois1_twitter
Hello Again! The best way to delete an image from Mrgeo, is it to use hdfs command line to delete it from hadoop hdfs? I can't find a specific mrgeo command to delete an image.
Dave Johnson
@djohnson729
Yes, that's correct
Josée-Anne Langlois
@jalanglois1_twitter
Ok, and it won't let a trace somewhere in Mrgeo? So I guess mrgeo is scanning a specific repository (mrgeo/images by default) in hdfs to know which images are available?! Is that correct?
Dave Johnson
@djohnson729
Yes the web services do that in order to respond to the GetCapabilities. The downside is that after deleting an image, the WCS GetCapabilities will still include it because the GetCapabilities is cached. It periodically re-reads the image.base to refresh the layers (configurable in mrgeo.conf). The WMS however does not cache the GetCapabilities (never got around to doing it), so that will re-scan each time.
In map algebra, you specify the name of the image you want to use, so if you delete the image, then the next time you run that map algebra, you'll get an error.
Josée-Anne Langlois
@jalanglois1_twitter
Ok thanks!
AmyZ
@AmyZ666
I'm a beginner.Why not update for so long?What's the difference between mrgeo and geotrellis?
AmyZ
@AmyZ666
hi?