Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Tino Desjardins
    @TDesjardins

    Hi guys, I've a fantastic client which has added all the layers of the getCapabilities into the layers parameter and it's performing a single getMap requesting the whole catalog.... I've the control flow but I can't find a parameter limiting this bad scenario.... is there any undocumented option (number of layers per request)?
    I think the WMS response size is not applicable since the size of the image is ok the problem is the number of resources needed to render the image...
    The geoserver is managing 1k layers so that request bring regularly to a kill of the tomcat (asking for too many resources/memory/connections etc)...
    It's an incredible easy way to let a geoserver down

    A possible solution is to use a proxy which could filter requests by parameter values. In your case that could be the number of layers in the layers param in a GetMap-request.

    Carlo Cancellieri
    @ccancellieri
    Thank you Tino, yes, indeed we are applying some contingency policy at the moment over the load balancer, but this is actually the first time I've seen such strange request (not tiled, all the layers in 1 request, huge image also...) so I was thinking to add (if missing this feature to the control flow.
    The risk to play with the load balancer is to bring geospatial concerns to the infrastructure team which may require, approvals, etc etc... we also really don't want to limit other clients which may correctly ask huge (but reasonable) requests.
    So I think the number of layers limit (get and post) is a good parameter that we would really like to add to the control flow letting him play with a blacklist of bad guys...
    Andrea Aime
    @aaime
    Yeah, if you find you really need a layer limit (still wondering about it) best to enforce it internally
    there are too many ways to do a request with N layer
    GET, POST form, POST xml, SLD_BODY, external SLD
    you don't want to have to parse a SLD in your proxy to figure out how many layers it has inside, I believe :-D
    Carlo Cancellieri
    @ccancellieri
    omg no, definitely... :D
    thank you all 4 thoughts
    Andrea Aime
    @aaime
    But as said... I'm skeptical... the renderer has two threads, one reads features from a layer and puts them in a bounded queue... the other reads features and renders them
    at any point in time the first is reading only one layer, and the other is rendering only one feature
    and the queue in between has fixed size, will block once full
    Carlo Cancellieri
    @ccancellieri
    I confirm a kill due to 12Gb of ram usage (machine limit) so at this point i think a memory leak is also playing it's role
    it's a cloud VM so I'm installing monitoring agents to see what happen to the ram usage (and when) then I'll check the dumps...
    Tino Desjardins
    @TDesjardins

    you don't want to have to parse a SLD in your proxy to figure out how many layers it has inside, I believe :-D

    Hehe, nope this isn't a good idea and that would also impact performance. My proposal is meant as a temporal solution which assumed that the "fantastic client" will request via HTTP GET. I could also imagine to extent the request limit settings in the WMS configuration of Geoserver (https://docs.geoserver.org/stable/en/user/services/wms/configuration.html) but suppose this is a rarely needed requirement.

    Jody Garnett
    @jodygarnett
    Could you put a memory limit on WMS requests; then this style of request should fail (as too much memory would be required in this case …)
    Carlo Cancellieri
    @ccancellieri
    Yep, I thought that, but the challenge is to check the biggest request, but definitely it make sense to have a limit... let me check we should have...
    Andrea Aime
    @aaime
    limit times the concurret requests allowed by control flow, and you get how much heap you need
    (well, add 1GB for the rest of GeoServer)
    Carlo Cancellieri
    @ccancellieri
    Good point, I'll do that in a different environment. the WMS limit was not set... :( thank you all :)
    Carlo Cancellieri
    @ccancellieri
    I set that even if it's also on the controlflow, anyhow it will probably not help so much, the estimated size of the rendered image is rarely reached the problem in this case is the number of touched resources and something which is not going well with the memory management which grows rapidly (i'm over geoserver 2.18.0 + adoptjdk11 and 'all' the 'recommended' settings for production).
    Jody Garnett
    @jodygarnett

    @jnh5y for the JTS upgrade for geoserver - it has some failure

    geoserver/geoserver#5258
    geotools/geotools#3617
    GeoWebCache/geowebcache#982

    Does this need to be done in conjunction with GT and GWC prs? I think you have a CI integration build showing the combo is successful.

    James Hughes
    @jnh5y
    only one of the GS jobs failed
    re-running it
    James Hughes
    @jnh5y
    the GS build has green checks now!
    Gotta continue my streak of 1 line PRs:)
    Jody Garnett
    @jodygarnett
    Thanks James, it is a minor update indeed (only changing version number)
    Is there anything for the upgrade instructions, perhaps how to enable/disable overlay ng ?
    2 replies
    Björn Harrtell
    @bjornharrtell
    Getting fairly inconsistent results attempting to build and test GeoServer with Java 17. But at least one dependency sticks out as especially problematic and that is cglib-nodep used in core stuff like GetFeature.java. cglib appears stagnant and depends on asm 7. Asm added JDK 17 support in version 9.
    Issue has been filed since April - cglib/cglib#191.
    Björn Harrtell
    @bjornharrtell
    Looks like the only place cglib is used is in GetFeature.java (https://github.com/geoserver/geoserver/blob/f732201d424c0aaf9545b2d0858016ea636450d5/src/wfs/src/main/java/org/geoserver/wfs/GetFeature.java#L705-L717) so perhaps that can be reworked.
    Andrea Aime
    @aaime
    @bjornharrtell that bit is an important element vs performance, it allows to lazily comput the feature count (something much complained and maligned about)
    a rework might need to rewrite the EMF generated classes.... which requires goat killing, the right moon, and the right version of Eclipse, and someone willing to use that abomination
    4 replies
    (not Eclipse per se, I'm talking about the EMF code generator)
    Björn Harrtell
    @bjornharrtell
    @aaime: it looks like it was somehow taken care of in springframework 5.3 (they import cglib with some quirks and things in place)... but upgrading to that looks like it needs servlet-api 4.0.1.
    Andrea Aime
    @aaime
    ugh
    which implies minimum version of Tomcat to use GeoServer to be 10 I believe
    and a bunch of import changes, at a minimum
    upgrading spring also brings an upgrade of spring-security, which we are strongly tied to, even on the internals... might be annoying
    I was also reading about wicket using cglib and having troubles.. wicket upgrades have been very painful so far
    Björn Harrtell
    @bjornharrtell
    Looks like Tomcat 9 is enough.
    As for Jetty, 10 is required.
    And well, didn't expect this to be easy :D
    Björn Harrtell
    @bjornharrtell
    Not sure but perhaps the use of cglib can be replaced with objenesis?
    Andrea Aime
    @aaime
    Maybe
    or maybe there is a newer cglib in the works with fixes? who knows
    Peter Rushforth
    @prushforth
    I opened a geoserver/geoserver#5259 that depends on a recently merged geotools change geotools/geotools#3620, and I'm not sure how to get the workflow to use the most recent gt-xml SNAPSHOT artifact. Can anyone give me a hint how to re-run the workflows with updated artifacts? Thanks!
    Andrea Aime
    @aaime
    @prushforth I actually restarted them a few hours ago
    Peter Rushforth
    @prushforth
    Me too, but the artifacts seem to be old e.g.: https://repo.osgeo.org/repository/snapshot/org/geotools/gt-xml/26-SNAPSHOT/gt-xml-26-20210818.092924-109.jar
    Peter Rushforth
    @prushforth
    I was able to reproduce the build error on my machine by building with gt-xml artifacts from the snapshot repo instead of my locally built repo. It seems the osgeo snapshot repo is not up to date with the geotools code, so there's nothing to be done on the geoserver side that will change that. I can just wait until the osgeo repo is updated. Thanks for your help!
    Andrea Aime
    @aaime
    The gt-main build has been broken for a number of days it seems
    Peter Rushforth
    @prushforth
    OK I reported it sort of on the geotools developer list. If it's broken I guess it's my duty :-)