by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Toni
    @t-book
    Hi @tsega_gitlab there is no support for netcdf afaik. Maybe you can convert it f.e. to csv or some of the other supported formats and upload it. Further I know there is a netcdf Plugin for geoserver. But never installed it.
    Tsegaselassie Tadesse
    @tsega_gitlab
    Hi @t-book , I think the second option is the way to go for me. Iโ€™m trying to install the NetCDF extension for Geoserver, it seems thatโ€™s all I need. Thanks for the help!
    adonis-albelda
    @adonis-albelda
    @t-book thanks body!
    adonis-albelda
    @adonis-albelda
    Hi all, I added route inside urls.py and access it in browser but it seems it does not take effect. How do I apply the changes of it ?
    @t-book can you help with this ? please
    adonis-albelda
    @adonis-albelda
    It changes only when I tried to run docker-compose up again. How can reload it without running docker-compose ?
    Albert van Niekerk
    @albertStaalburger
    My understanding is that if you change any settings in the conf files, you will need to run docker-compose up -d
    Be careful! The following command drops any change you might have done manually inside the containers, except for the static volumes.
    Albert van Niekerk
    @albertStaalburger
    I am having trouble figuring out why I cannot import layers, the UI simply reports Unauthorised, and in the logs I see:
    Error creating import session
    Traceback (most recent call last):
      File "/usr/src/geonode/geonode/upload/upload.py", line 306, in save_step
        next_id = _get_next_id()
      File "/usr/src/geonode/geonode/upload/upload.py", line 229, in _get_next_id
        importer_sessions = gs_uploader.get_sessions()
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 47, in get_sessions
        return self._call(self.client.get_imports)
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 34, in _call
        robj = fun(*args)
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 221, in get_imports
        return parse_response(self._request(self.url("imports")))
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 198, in _request
        raise RequestFailed(resp.status, content)
    gsimporter.api.RequestFailed: (401, b'<!doctype html><html lang="en"><head><title>HTTP Status 401 \xe2\x80\x93 Unauthorized</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 \xe2\x80\x93 Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Unauthorized</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.20</h3></body></html>')
    request failed
    Traceback (most recent call last):
      File "/usr/src/geonode/geonode/upload/views.py", line 677, in view
        resp = _steps[step](req, upload_session)
      File "/usr/src/geonode/geonode/upload/views.py", line 201, in save_step_view
        charset_encoding=form.cleaned_data["charset"]
      File "/usr/src/geonode/geonode/upload/upload.py", line 409, in save_step
        raise e
      File "/usr/src/geonode/geonode/upload/upload.py", line 306, in save_step
        next_id = _get_next_id()
      File "/usr/src/geonode/geonode/upload/upload.py", line 229, in _get_next_id
        importer_sessions = gs_uploader.get_sessions()
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 47, in get_sessions
        return self._call(self.client.get_imports)
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 34, in _call
        robj = fun(*args)
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 221, in get_imports
        return parse_response(self._request(self.url("imports")))
      File "/usr/local/lib/python3.7/site-packages/gsimporter/client.py", line 198, in _request
        raise RequestFailed(resp.status, content)
    gsimporter.api.RequestFailed: (401, b'<!doctype html><html lang="en"><head><title>HTTP Status 401 \xe2\x80\x93 Unauthorized</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 \xe2\x80\x93 Unauthori
    Can this issue be because of file access? The request has not been applied because it lacks valid authentication credentials for the target resource.
    Toni
    @t-book
    @adonis-albelda urls.py will be converted as pyc. You could try to log into the django container and delete the pyc file which should be regenerated and respect your changes. Another helpful trick If urls do not resolve as you expect is to set debug true and call some not existing url. Django will tell you which urls it knows of. Last but not least. I would suggest to develop with paver and push to production when ready.
    And if really needed you could also only up the Django service again.
    adonis-albelda
    @adonis-albelda
    @t-book , whats the exact file name of that pyc file ?
    Toni
    @t-book
    @albertStaalburger What happens when you try another importer? https://github.com/GeoNode/geonode/blob/master/geonode/settings.py#L927 does it work?
    Toni
    @t-book
    anybody experienced problems regarding the following migration? https://github.com/jazzband/django-taggit/blob/master/taggit/migrations/0003_taggeditem_add_unique_index.py#L12-L14 (run in a key error from a 2.10 database to gn3 ...)
    Scott McIntosh
    @smcintosh-icf
    I managed to create a new install of 3.0 via the doc that @afabiani created. The install went smoothly and everything came up. However, now I want to use my production db, which wasn't addressed in the documentation but the properties in the .env file seemed intuitive. But I don't think it's working as no tables get created (and I'm only assuming they would - maybe there is a db script?).

    Here are is the section of .env
    GEONODE_DATABASE=geonode
    GEONODE_DATABASE_PASSWORD=<secret>
    GEONODE_GEODATABASE=geonode_data
    GEONODE_GEODATABASE_PASSWORD=<secret>

    DATABASE_URL=postgres://geonode:<secret>@postgres.cb9sgrxx87k8.us-east-1.rds.amazonaws.com:5432/geonode
    GEODATABASE_URL=postgis://geonode:<secret>@postgres.cb9sgrxx87k8.us-east-1.rds.amazonaws.com:5432/geonode_data

    Toni
    @t-book

    @smcintosh-icf you're using docker, right? never wired the container with an external db. In any case I think as django might has set the db service as a dependency in docker-compose you would have to start it. I would suggest starting with small steps. First of all I would test if all works fine with your connection and the used docker network. Unsure if the django container has postgres-client installed if not you could test with something like:

    docker exec -it django4your_geonode bash
    apt-get update
    apt-get install -y postgresql-client
    psql --host=postgres.cb9sgrxx87k8.us-east-1.rds.amazonaws.com --port=5432 --username=geonode -c "SELECT 'Connection works;'"

    After that continue do update your env. file

    Scott McIntosh
    @smcintosh-icf
    @t-book yes, I'm using docker. I can test the connection easily. Mostly I was unclear on the URL format. I think I have the format correct (user:pwd@host etc.), although I did wonder why the password was supplies in the url if its set in the password properties.
    Toni
    @t-book
    ahm a password in the connection string :/ only thing I know where this happens is with basic auth. But maybe it is expected. You get something meaningful in startup log of django?
    like connection not successful ... ?
    Scott McIntosh
    @smcintosh-icf
    Connect worked fine. postgres client was already there.
    Nothing in the log that looked like connection attempt.
    Scott McIntosh
    @smcintosh-icf
    Let me look
    Scott McIntosh
    @smcintosh-icf
    Oh, yeah I did look at that.
    Toni
    @t-book
    the invoke tasks come from tasks.py . you should see that fixtures and migration run in your startup logs
    what is currently done should be printed like: https://github.com/GeoNode/geonode-project/blob/master/tasks.py#L194
    Scott McIntosh
    @smcintosh-icf
    Thanks. I'll look at logs some more
    Toni
    @t-book
    maybe only follow django logs : docker-compose logs -f django
    Scott McIntosh
    @smcintosh-icf
    ok
    Toni
    @t-book
    if your stuck you could even change tasks.py to gather more information about your connection. it's all python. good night ;))
    Scott McIntosh
    @smcintosh-icf
    @t-book Got it working. invoke.log was helpful.
    First issue was Postgres-client was in the container, but not in the host (which performs the migration).
    Second issue that came up was it always tried to use nysdot_geonode and nysdot_geonode_data even though I had geonode and geonode_data in .env properties. So I gave up and renamed my database to nysdot_geonode and nysdot_geonode_data and it came up.
    Though you'd like to know :)
    Toni
    @t-book
    Congrats! Great to hear!
    Scott McIntosh
    @smcintosh-icf
    Thanks. Now to get the external data moved to a mounted filesystem...
    Tek Bahadur Kshetri
    @iamtekson
    I am planning to add around 40 thousand rasters into geonode for data visualization. Each raster having an average size of 300mb. These rasters may differ from the different time periods, crop type, Reflected Ceiling Plan. I am afraid due to this huge amount of data will get crashed. @afabiani @t-book could you please suggest me how can I store these rasters so that the GeoNode won't get crashed?
    I already asked a question in StackOverflow (https://gis.stackexchange.com/questions/363579/big-data-with-geoserver) but no reply there, so I am asking here. Please suggest a better idea.
    Toni
    @t-book

    @iamtekson It is strongly advised to preprocess the raster files! First test jpeg compression and YCbCr color model on a subset of your data. Keep in mind that you might have to handle 1 and 2 band next to raster images with an alpha channel as well. This should help you getting started: https://geoserver.geo-solutions.it/edu/en/raster_data/processing.html

    Besides of lowering your image sizes and f.e. block sizes you should think about future data migration, backups etc. Last but not least do not forget. sync_permissions (geonode <> geoserver) is a slow process 2300 layers took around 4 hours yesterday for me. So be prepared 40k will take some time!

    (in case you import them in geoserver first and sync afterwards ...)
    Tek Bahadur Kshetri
    @iamtekson
    Thank you for the suggestion. My rasters are GeoTIFF files. some of them are continuous rasters and some of them are classified raster. I don't care how much time does it takes for the first time setup. What if I uploaded these rasters to GeoServer using Geoserver rest API? And after that how can I list out the layers in GeoNode?
    Tek Bahadur Kshetri
    @iamtekson
    My main concern is about will the GeoServer get crashed due to big data or not? Does it support such a huge files?
    Alessio Fabiani
    @afabiani
    @smcintosh-icf if you could find some time/resources to improve the docs with your findings, that would be great
    Toni
    @t-book
    @iamtekson As said I would suggest testing on a subset and see how things work with your server performance. In my opinion, the mentioned preprocessing is essential. For example, 346gb of geotiffs shrink to 46 GB in a recent job.
    regarding performance, the first thing which would come to my mind is scaling but since geoserver afaik does not scale horizontally your only chance is vertical scaling, optimization of data, and configuration. (A bit older https://de.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 but still an interesting read). By conscientious testing, you might find bottlenecks before spending days or weeks trying to handle 40k of raster files. In case you directly add your tiffs to geoserver (which I would do as well) you can use https://docs.geonode.org/en/master/admin/mgmt_commands/index.html?highlight=updatelayers#management-command-updatelayers
    Last but not least I do remember a geonode instance with ~80k of layers that had getCapabilies disabled to not stress the server with something it cannot handle. Not much but hope this helps.
    Scott McIntosh
    @smcintosh-icf
    @afabiani As soon as I get everything up then I can do that. I'm making good notes as I go. But right now I've just started testing and discovered I can't upload a layer, so I need to sort that.
    Have you come across this error?
    Client Error: Bad Request
    Unexpected exception 'NoneType' object has no attribute 'store_type'
    Was simply uploading a small GeoJSON file. Upload went all the way, then the error.
    Toni
    @t-book
    @smcintosh-icf does your geoserver has access to your geonode_data db as well?
    Tek Bahadur Kshetri
    @iamtekson
    Wonderful โฃ๏ธ๐Ÿ˜Š. Thank you very much for your kind suggestion and documents link. As you mentioned about the geonode with ~80k layers, that motivated me a lot to visualize these rasters with geonode. I will try to follow your suggestion and if any problem occur, i will let you know. Thank you very much for quick response.๐Ÿ˜Š๐Ÿ˜Š
    Scott McIntosh
    @smcintosh-icf
    @toni I didn't think to check. I'll look tomorrow and let you know.
    Meant for @t-book ...