jmespath/chat

Florent Gallaire

@fgallaire

but I prefer to ask experienced people

glenveegee

@glenveegee

I've tried variations on this a few times and I always seem to think I can somehow do it. Unfortunately I don't believe it's possible because jmespath cannot set keys of object projections using values from the current context (as far as I know... someone please correct me if I'm wrong)

What language are you hoping to use jmespath in because there are spec compliant libraries out there that do let you do this fairly easily

I know of a few javascript/typescript libraries out there that extend the jmespath spec to include this functionality

Florent Gallaire

@fgallaire

thanks a lot for your tries, yes I think jmespath needs well chosen lists to be efficient, but I prefer to avoid lists for some reasons (less repetitions and best UI to edit dicts)
I use Python

JensOAndersen

@JensOAndersen

Hey people, i hope this is the correct place to ask, i'm trying my hand with jmespath, and i've ran into some null value issues, i've tried looking around on stackoverflow and the documentation, but i havnt been able to find any solution to my issue:
I've got the following:

{
    "items": [{
        "val1": "some",
        "val2": null
    }, {
        "val1": "other",
        "val2": "value"
    }]
}

To which i apply the transformation:
items[*].{newValue:join(' - ',[val1, val2])}
However, it doesnt work, as items[0].val2 has the value 'null', is there some way to look through properties and replace null values with a default value, emptystring in this case.

varadasandeep

@varadasandeep

trying to filter output of az pipelines variable-group variable list but failing in all ways

Name Is Secret Value

bearer_token False
org_id True
org_id1 False dhsakjfhakjshfkjdahf
owner_id True
owner_id1 False mdfjsadhfkjsdahfkjsdhf
password True
password1 True
projectid False
username False testtest
username1 False test

need the Name column

Jan Walzer

@jwalzer

Hi! I need some help with a json query. My data (simplified and to yaml) is

network:
  ipfabric:
    lab:
      links:
        - nodes:
            "h01":
              interface: "swp1"
              ip: 192.168.253.180
            "h02":
              interface: "eth0"
              ip: 192.168.253.250
          networks:
            ip4:
              cidr: 192.168.253.0/24
          bfd: yes
          bgp: yes
        - nodes:
            "h01":
              interface: "swp2"
              ip: 192.168.23.180
            "h04":
              interface: "eth2"
              ip: 192.168.23.250
          networks:
            ip4:
              cidr: 192.168.253.0/24
          bgp: yes

I'm querying for all interfaces on "h01" with ipfabric.*.links[*].nodes.h01.interface[] which works fine ... But:

I want now to query for "all interfaces on h01 where bfd is enabled"

... I tried with different variations on ipfabric.*.links[? bfd == `yes`].nodes.h01.interface[]

can somehone point me to the direction?

Oryulin

@oryulin

Based on your sample data structure above, I "think" this is what you were hoping for:

ipfabric.lab.links[?bfd == `yes`].nodes.h01.interface

Since I do most of my testing in Ansible, one note I will add is that yes translates to True, so I had to wrap the yes in single quotes to stop it from processing as a boolean value. If you are using the same, you can change it to this to search for the Boolean True instead of the string yes:

ipfabric.lab.links[?bfd == `true`].nodes.h01.interface

Jan Walzer

@jwalzer

Yes, I'm also using this in ansible and I thought already of the yes->true transition, but I'm missing practice on json_query and so I wanted some opinions and help. I'll give it a try

the '*' instead of 'lab' is correct, because there are other structures overlapping with different names

Oryulin

@oryulin

Since you are using Ansible as well, here is my "testing" task:

- name: JMESPath Testing
  vars:
    jmespath_query: >-
      ipfabric.lab.links[?bfd == `true`].nodes.h01.interface
  debug:
    var: network | json_query(jmespath_query)

Oryulin

@oryulin

Re-reading what you wrote, I think I am gathering the picture. lab also has other similar environments that need to be searched for thus needs to be wildcard'ed if I am understanding right...trying this:

ipfabric.*[].links[?bfd == `true`].nodes.h01.interface | []

The thing about using the dictionary wildcard ipfabric.* converts the what you would expect to be dict results, actually converts it to a list. You can tell by doing this:

ipfabric.*

You will notice the top results is a list now, instead of a dictionary. To get past this conversion I simply flatten the list result from the asterisk and keep going. At the end you will notice that there is a list within a list, because of the flattening...so I pipe to another flatten and thus I think is the end goal you were hoping?

Jan Walzer

@jwalzer

ah ... your explaination makes sense ... I'm trying this

Jan Walzer

@jwalzer

BTW: I'm currently testing quite the same way as you do in ansible, so I'm probably not completely on the wrong track ;)
so my current reasoning is: I'll try to match the "true" and I get an empty list. I'll try to match the negative (bfd != true) and I also get an empty list. So somehow it doesn't return anything after the filtering. I cut down from the end to see if I miss something.

and it seems, one of the culprits is the convertion between "yes" and any kind of "true" - For testing purposes I changed my input data to literal true - and then I can in jmespath compare to true - but the same does not work for yes

Oryulin

@oryulin

It seems like you found your issue but I had already typed this out, and it is a helpful gem for new users and testing so I sent it anyways:

For JMESPath testing, I always start at the beginning and make my way down the rabbit hole, so to speak.
Start by commenting out the | json_query(jmespath_query)

Then if that works, uncomment it and start cutting down the jmespath_query filter one step at a time...like so:

ipfabric.
ipfabric.*
ipfabric.*[]
ipfabric.*[].links
ipfabric.*[].links[?bfd == `true`]
ipfabric.*[].links[?bfd == `true`].nodes
ipfabric.*[].links[?bfd == `true`].nodes.h01
ipfabric.*[].links[?bfd == `true`].nodes.h01.interface
ipfabric.*[].links[?bfd == `true`].nodes.h01.interface | []

If you don't see results from the first comment, then you know the variable network is not the right one you should use in yours...look at where your data is coming from and change accordingly. If its in the jmespath filter, you will see it break (empty results) eventually and know where your query needs to be updated.

Jan Walzer

@jwalzer

and I just now found out, that: If its a boolean (yes) then I don't need to compare to anything, but can just do an ipfabric.*[].links[?bfd].....

@oryulin yep, thats how I normally construct this stuff to. This time I had to adjust the existing(and working) query for that new parameter, and thought, it could be so easy. But it was an insightful chat, nevertheless ... thank you

Oryulin

@oryulin

Absolutely @jwalzer, glad I could be of assistance before my work day started.

Jan Walzer

@jwalzer

hehe .. here in Berlin the workday is about to end soon ;) -- but to summarize my understanding: Because of the flattening (that I'm doing nevertheless at the end) my old query worked without the filtering. But because of the .. thing flattening also the filter didn't work, and I had to change to .[]. to avoid that flattening. Right?

Oryulin

@oryulin

Your original query was accurate in my opinion...I think the only problem you had was due to ipfabric.*.links as that would never yield results because the asterisk converts the dict to a list of matching dicts... so we had to deal with that by adding the flattener right after the asterisk:

ipfabric.*[].links

The rest was on par I believe (minus of course Ansible's fun yes/no to boolean good times). :)

You could also pipe it again if that makes it look more visually appealing too, semantics to me.

ipfabric.* | [].links[?bfd].nodes.h01.interface | []

Jan Walzer

@jwalzer

I have to admit, I'm not yet so natural with the syntax of jmespath and I'll need more practice on that before the semantics are intuitive for me. But at least I could solve that problem, and I'm sure I'll get to the power of jmespath in the future again, before trying uggly things with loops in ansible/jinja

Oryulin

@oryulin

Not that this is the write chat for this statement, but JMESPath > Ansible loops in the sense of speed and efficiency. If you are only data processing/manipulation, I would stay FAR away from Ansible loop as its resource costly/timely. JMESPath can filter hundreds of thousands of list/items in seconds whereas that same processing in Ansible loop...you'll be sitting a while. If you are doing actual work like API calls, then you sort of have to do Ansible loop.

Jan Walzer

@jwalzer

Jep, thats no question at all -- I've seen lots of uggly ansible code with recursive loops down the hierarchy, just to find some objects or keys in data structures, that are possible with a oneliner, exactly like the one above. Luckily, Infrastructure managed with ansible isn't that ofthen so large, that the timing issue really matters, but of course, it's also about complexity and maintainability of the code, and a single, readable oneliner is much better in that metric to

Oryulin

@oryulin

@jwalzer -
Be sure to use this little nugget with JMESPath and Ansible if you ever attempt to use functions in JMESPath with Ansible such as map, ends_with, et...:

    - name: JMESPath Testing
      vars:
        jmespath_query: >-
          ipfabric.* | [].links[*].nodes.h01.interface | [] | [?ends_with(@,'1')]
      debug:
        var: network | json_query(jmespath_query)

By all accounts this should work but you find that you get a "type" error:
In function ends_with(), invalid type for value: swp1, expected one of: ['string'], received: \"unknown

This is due to Ansible's data types not matching the JSON spec is what I believe is the root cause. To fix it simply add the to_json and from_json syntax before the json_query and profit.

    - name: JMESPath Testing
      vars:
        jmespath_query: >-
          ipfabric.* | [].links[*].nodes.h01.interface | [] | [?ends_with(@,'1')]
      debug:
        var: network | to_json | from_json | json_query(jmespath_query)

Jan Walzer

@jwalzer

this must be quite a "LolWTF!?!?!"-Moment then *g

and these are the people, telling me, that perl code is uggly

Christoph

@cruepprich

Hi all. I have a question regarding querying time stamps. In the example below I would like to filter the items that are within the last 10 days of the current date. Is there a JMESPath function/feature that allows me to do that?
So when I run the query on September 16th it should return the first item only (backup b3).

[{
    "backup": "b3",
    "time_ended": "2020-09-10T14:55:39.840000+00:00"
}, {
    "backup": "b2",
    "time_ended": "2020-08-25T15:40:48.740000+00:00"
}, {
    "backup": "b1",
    "time_ended": "2020-07-10T15:40:48.740000+00:00"
}]

alonsocamaro

@alonsocamaro

Hi everybody

where can find a jmespath online expression evaluator

I find jsonpath ones but not jmespath ones

more precisely, the one in jmespath.org doesn't always update when fields are updated

so I'm looking for another one

Oryulin

@oryulin

@alonsocamaro - THe homepage for JMESPath can be used as an online evaluator if desired. Its a real-time test and you can manipulate the search and json structure that will be searched.

https://jmespath.org/

I have not ran into an issue where it has "failed" for me thus far unless there is an error in my syntax which is most likely the cause of your not updating. That is one area where it lacks is it doesn't let you know when the syntax has an error so its likely assumed to be a web-based error. If you have a specific scenario/filter feel free to type it here and if I am paying attention will attempt to help.

Charles Albrecht

@karillo_gitlab

I asked this on stackoverflow as well, but thought I'd ask it here to try and get some traction.

AWS's list-secrets has an odd way of listing version strings as keys:

{
  "SecretList": [
    {
      "Name": "sandbox_config",
      "Description": "sandbox config file",
      "SecretVersionsToStages": {
        "2895f1a4-bcae-46b2-9a45-4f06f490d8ed": [
          "AWSCURRENT"
        ],
        "336a5030-cd81-4626-a100-1aa68b70b5b1": [
          "AWSPREVIOUS"
        ]
      }
    }
  ]
}

I'm trying to figure out a query that will let me pull out the versionkey where AWSCURRENT is in the value array.

This seems like it's close:

SecretList[].{Version:SecretVersionsToStages.*[?@==`AWSCURRENT`]|[]|[0] }

But I'm clearly missing something, since that just spits back AWSCURRENT to me.

PeiSong

@naivefun

Hi, how can I do simple math with jmespath? like length(values(@)) * 2 ?

glenveegee

@glenveegee

Can't be done with the standard jmespath spec. Some "augmented" jmespath libraries add this functionality but not available in all languages

PeiSong

@naivefun

oh I need use custom func to impl myself

glenveegee

@glenveegee

There are libraries out there that provide the ability to add your own custom functions to the existing jmespath spec. I'm afraid this is the only way I have found to make it possible to support the kind of operations I've required in the past.

I've also created proposals as suggested by the author @jamesls here (https://github.com/jmespath/jmespath.jep) in the hope that they get added to the spec

Unfortunately @jamesls seems quite busy at the moment as there hasn't been any feedback from him in the last 6 months

glenveegee

@glenveegee

If you could... would you want to be able to do something like this:

expect(
    search([6.1, 4.2, 6.3], "_groupBy(@, as_lambda('Math.floor'))")
).toStrictEqual({
      '4': [4.2],
      '6': [6.1, 6.3],
    });

or perhaps

expect(
    search([1, 2], "_flatMap(@, as_lambda('x => [x, x]'))")
).toStrictEqual([1, 1, 2, 2]);

Thoughts?

Where communities thrive

People

Repo info

Activity