Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    hell-racer
    @hell-racer
    https://jmespath.org/contents.html
    "JSON Matching Expression paths"
    Nolan Woods
    @innovate-invent
    the authors name is also James
    Maxime Labelle
    @springcomp
    @/all we are making good progress trying to standardize some improvements to JMESPath. Please, feel free to share your feedback and comments.
    Stephane Landelle
    @slandelle

    Comparisons between numbers and strings is currently undefined in the grammar as ordering operators are only valid between two numbers. However, popular implementations do make some attempt to cast the string to a number before performing the comparison, so it mostly works even though it is not standards-compliant.

    If you guys are considering JMESPath 2.0, should this String comparison be made a standard?

    Maxime Labelle
    @springcomp
    @slandelle I have specified your request. However, I’m not sure I agree with implicit conversions, after all...
    msouilhi
    @msouilhi
    I would extract the keys for this dictionnary to get kas_sub.med , kas_sub.test
    {
     "FACTS_ENTRY": {
            "kas_sub.med":  "true",
            "kas_sub.test":  "true"
    }
    Nolan Woods
    @innovate-invent
    msouilhi
    @msouilhi
    @innovate-invent i am using it inside a playbook in ansible and i am new to jmespath so i would use it like this keys(FACTS_ENTRY)?
    hell-racer
    @hell-racer
    Yep
    (probably you've tried it at https://jmespath.org/ and it didn't work because you missed one closing curly brace in your input json)
    msouilhi
    @msouilhi
    thank you so much guys i will try it right now
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    I've got a JSON output that looks like the (simplified) below:

    {
    "Code": 1000,
    "GlobalServers": [
    {
    "Name": "bar",
    "Country": "IS",
    "Status": 1,
    "Servers": [
    {
    "Name": "foo",
    "Status": 1
    }
    ]
    }
    ]
    }

    But I am struggling to filter it.
    GlobalServers[?Status==1].... works
    but then if I try
    GlobalServers[?Status==1].Servers[?Status==1] ...I get no ouput

    hell-racer
    @hell-racer

    Hi! At this point

    GlobalServers[?Status==`1`].Servers

    ... you have an array of arrays:

    [
      [
        {
          "Name": "foo",
          "Status": 1
        }
      ]
    ]

    So you have to flatten it and stop the projection before filtering:

    GlobalServers[?Status==`1`].Servers[]|[?Status==`1`]

    Output:

    [
      {
        "Name": "foo",
        "Status": 1
      }
    ]
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    I see, that makes sense. Thanks for the clear explanationhell-racer (hell-racer) . So I guess the only issue then is that if I flatten it, I loose the ability to get a key value from the parent ? (e.g. if I wanted to pull "Country" value from the parent in addition to the data from the filtered child nodes)
    hell-racer
    @hell-racer
    It could be possible to filter while keeping nesting level. Just post exact shape of output you need (based on input you've posted), and I'll try to come up with the query.
    hell-racer
    @hell-racer
    If you need just the Servers and Country:
    GlobalServers[?Status==`1`].{ Servers: Servers[?Status==`1`], Country: Country }
    If you need everything while filtering both levels:
    GlobalServers[?Status==`1`].merge(@, { Servers: Servers[?Status==`1`] })
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Sorry for the delay in reply @hell-racer , I'll go test these and let you know.
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    @hell-racer: The suggestion with merge(@) worked great. Thank you again !
    hell-racer
    @hell-racer
    nice :)
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]

    @hell-racer: There's one more thing, I was hoping to find the answer myself but I can't find the right way to do it. Building on your "GlobalServers[?Status==1].merge(@, { Servers: Servers[?Status==1] })".....

    The GlobalServers parent node has a "score" value for each entry, I would like to only return the node for the highest score (that also matches the filter).

    So far the nearest I have managed to get is this:
    "max_by(GlobalServers[],&Score).merge(@, { Servers: Servers[?Status==1] })"

    But no matter how I try to integrate the original GlobalServers filter, I can't manage do to it. I've tried putting max_by() before the filter, after the filter, inside the filter ... jmespath complains each time.

    I'm sure I'm missing something simple !
    hell-racer
    @hell-racer
    Can you post source data with all the fields involved?
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    @hell-racer: Sure. Lucky for you it's coming from a public API, so fill your boots ... https://api.protonmail.ch/vpn/logicals
    hell-racer
    @hell-racer

    It returns ERR_CONNECTION_RESET :(.
    Ok, on the site https://jmespath.org/ given input:

    {
        "Code": 1000,
        "GlobalServers": [
            {
                "Name": "bar1",
                "Country": "IS",
                "Score": 1,
                "Servers": [
                    {
                        "Name": "foo1",
                        "Status": 1
                    },
                    {
                        "Name": "foo2",
                        "Status": 2
                    }
                ]
            },
            {
                "Name": "bar2",
                "Country": "US",
                "Score": 2,
                "Servers": [
                    {
                        "Name": "foo",
                        "Status": 1
                    },
                    {
                        "Name": "foo2",
                        "Status": 2
                    }
                ]
            }        
        ]
    }

    Your query:

    max_by(GlobalServers, &Score).merge(@, { Servers: Servers[?Status==`1`] })

    Returns:

    {
      "Name": "bar2",
      "Country": "US",
      "Score": 2,
      "Servers": [
        {
          "Name": "foo",
          "Status": 1
        }
      ]
    }
    So if it doesn't work in your environment, probably there is a problem with specific JmesPath implementation.
    Ah, maybe you have unnecessary [] in the GlobalServers[].
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    That's interesting. I just tried it over a "private" browser window and it came straight up. Do you get the same with a slightly different url: https://account.protonvpn.com/api/vpn/logicals ?
    hell-racer
    @hell-racer
    In this case GlobalServers is already an array (flat array), so you don't need to flatten it (neither start a projection).
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Regarding your answer, yes "max_by(GlobalServers, &Score).merge(@, { Servers: Servers[?Status==1] })" works. But that's not my problem. I still need to integrate the original "GlobalServers[?Status==1]" parent filter and I can't find a way to do that in conjunction with max_by()
    hell-racer
    @hell-racer
    So filter by status then select max by score?
    That makes sense )
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Yeah, I already tried that construct (if you're thinking "GlobalServers[?Status==1].max_by(GlobalServers, &Score).) or something like that
    In terms of concept yes ... not much point picking a high-scoring dead server. ;-)
    hell-racer
    @hell-racer
    max_by(GlobalServers[?Status == `1`], &Score).merge(@, { Servers: Servers[?Status==`1`] })
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Now that's one of the hundreds of combinations I didn't try !
    ;-)
    hell-racer
    @hell-racer
    :)
    Another way would be:
    GlobalServers[?Status == `1`]|max_by(@, &Score).merge(@, { Servers: Servers[?Status==`1`] })
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Yeah, that seems to run great with my static test JSON.
    Regarding your second suggestion of the pipe, I'm sure I tried that, but maybe my syntax was slightly off.
    hell-racer
    @hell-racer

    The problems with

    GlobalServers[?Status==1].max_by(GlobalServers, &Score) ...

    ... are these:

    1. After initial filtering you're in a projection, e.g. if you add . after that, it will evaluate everything after the . in context of single element of the filtered array, so max_by will treat a single item as array and try to find max in it. So you have to stop the projection, to allow the next function to operate on the entire filtered array.
    2. Now if you change . to | you are now in context of the filtered array of GlobalServers, so you have to use @ to refer to it instead of GlobalServers.
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    Thanks for the nice clear explanation. Much appreciated @hell-racer jmespath is a wonderful tool, but also devilishly tricky to learn !
    hell-racer
    @hell-racer
    But once you learn it you feel the power like a wizard :)
    hansenjohansen
    @hansenjohansen:matrix.org
    [m]
    In some respects it would be nice if the docs had a couple of "real-world" examples with slightly trickier syntax rather than just the simple one-liners based on simple tasks.
    At some point I even considered running jmespath twice for my application (once to save "max" to a variable and the second to use it in the filter). But I thought surely there must be a way to do it as one, hence I came back to ask the wizards here !
    hell-racer
    @hell-racer
    I think projections is most difficult topic and definitely needs more examples.
    Maxime Labelle
    @springcomp

    In some respects it would be nice if the docs had a couple of "real-world" examples with slightly trickier syntax rather than just the simple one-liners based on simple tasks.

    Thanks for the feedback. We will try and update the spec accordingly.

    I think I have a mental model that works for explaining what are projections and how they work.

    Imagine a JMESPath expression as a pipeline, like a series of more focused expressions that process input and produce output.
    While the input and output of the whole JMESPath expression is always a JSON token, what flows through the pipeline can be thought of as either of two things:

    • A JSON token
    • A "projection"

    I define a "projection" as being a list of JSON tokens (very similar to a JSON array).
    But the difference is, while a JSON array is passed as a single value to the downstream expression in the pipeline, a "projection" sees its elements branch off individually to the downstream expression.

    The only way to fold a projection back to a regular JSON array is to either use a | or to exit the pipeline. That is, if the result of the evaluation ends with a "projection", the JMESPath evaluation engine folds the result to JSON array.

    So if you use a hash-wildcard-expression on an input JSON array, the result will be a projection, that is, a list of JSON elements, each of which will go through the downstream expression.
    Maxime Labelle
    @springcomp

    Consider the following input JSON document:

    [
      { "name": "jack" },
      { "name": "john" }
    ]

    Consider the following very similar expressions:

    • search( [*].name.length(@), …) -> [4, 4]
    • search( [*].name|length(@), …) -> 2
    1. The input of both entire expressions are a JSON array.
    2. After evaluating the first expression in the pipeline, [*], the output is a projection, i.e a list containing two JSON objects that must be projected to the downstream expression name.
    3. Evaluating the identifier name happens twice, once for each element in the projection, and returns a new projection, i.e a list of two JSON strings.
    4. Now, that’s where the two expressions differ.

    Using . in a sub-expression

    • When using a . in sub-expression, the two JSON strings in the projection are projected onto the length(@) expression.
    • Each JSON string is sent as the first argument to the length() function which outputs a JSON number.
    • Both JSON numbers are aggregated, the result being yet another projection, ie a list of two JSON numbers.
    • Since this is the end of the entire expression, the JMESPath evaluation engine folds the projection to a regular JSON array.

    Using |in a pipe-expression

    • When using a | in a pipe-expression, however, the projection is folded back to a regular JSON array immediately.
    • The JSON array contains two JSON strings, each of which is the value for the name properties in the original JSON document.
    • This array is passed as the first argument to the length() function, which returns a JSON number.
    • Since this is the end of the entire expression, that is what the JMESPath evaluation engine returns.