kaitai_struct/Lobby

oldbug

@oldbug:matrix.org

[m]

ok ok

target is python

oldbug

@oldbug:matrix.org

[m]

yup, looks good:

    class Callsign(KaitaiStruct):
        def __init__(self, _io, _parent=None, _root=None):
            self._io = _io
            self._parent = _parent
            self._root = _root if _root else self
            self._read()

        def _read(self):
            self.callsign = (self._io.read_bytes(6)).decode(u"ASCII")
            if not  ((self.callsign == u"ALL   ") or (self.callsign == u"WH6GEH")) :
                raise kaitaistruct.ValidationNotAnyOfError(self.callsign, self._io, u"/types/callsign/seq/0")

Mikhail Yakshin

@GreyCat

:+1:

We should finally get this valid stuff documented some day...

oldbug

@oldbug:matrix.org

[m]

yeah, I might try to help on that...

I'm busy with some other stuff actually, but may be able to offer some time in a few days, probably weeks

Mikhail Yakshin

@GreyCat

That would be really cool ;)

oldbug

@oldbug:matrix.org

[m]

this valid: things are bringing us a lot further in means of sanity checks when feeding from obviuously wrong sources...

The Dread Vixen Alinsa

@alinsavix

That'd be a useful thing!

Also would be useful: being able to get at user-defined metadata for a given struct/field from the final runtime (what the compiler outputs). So that I could, for example, add a bit of metadata to a field that says how to display that field, that my app could read that for a given field when it comes time to display it.</random>

Petr Pučil

@generalmimon

@GreyCat

We should finally get this valid stuff documented some day...

Sure, but in the meantime I refer to the relevant issue kaitai-io/kaitai_struct#435, because everything a user needs to know is contained in the first comment. But I'm aware that it is not properly documented and I've been hoping to make it happen someday.

%) Check if that's still implemented end-to-end
I vaguely recall that it might be implemented in some languages but not the others

Um, don't we have the CI dashboard just for checking the implementation status of some feature among target languages? Why the uncertainty?

The test ValidFailAnyofInt looks just as green as the other ValidFail* tests, so I don't expect any major problems here. In Nim, Perl and Construct, validations are not yet implemented at all.

But yeah, there are several valid-related and other bugs that I know of and I have not added tests or created issues for them, I guess I need to make up for that ASAP...

oldbug

@oldbug:matrix.org

[m]

Oh, I'm really happy with the current state of it!

Curtis Mechling

@cmechlin

need to access elements from sibling structure, meaning struct A needs elements from struct B, where A & B are not derived from the same template structure, but share the same parent struct. is this possible?

Mikhail Yakshin

@GreyCat

@cmechlin It depends. Is there something like you need to access a neighbor member of the array?

Curtis Mechling

@cmechlin

there is an array of ascii values in the neighbor. the first struct contains the index to the array in the second struct, so want to "lookup" the ascii in the 2nd and store its value in the first 1st

Mikhail Yakshin

@GreyCat

Can you show a sample KSY?

Curtis Mechling

@cmechlin

yes. give me a min

Curtis Mechling

@cmechlin

i think this is the captures what i want to do. struct_a contains array of elements w/ ascii values w/ also an index field, struct_b contains array of indexs. want to use index in struct_b to grab ascii from struct a where indexs match and store in struct_b.

meta:
  id: test
  file-extension: bin
  endian: be

seq:
  - id: sections
    type: cfg_sections
    size: section_size
types:
  cfg_sections:
    seq:
      - id: sections
        type: section
        repeat: eos

  section:
    seq:
      - id: header
        type: header
      - id: body
        type: 
          switch-on: header.type
          cases:
            'struct_types::STRUCT_A': struct_a
            'struct_types::STRUCT_B': struct_b
            _: unknown

  struct_a:
    seq:
      - id: num_elements
        type: u2
      - id: records
        type: record
        repeat: expr
        repeat-expr: num_rows

  record:
    doc: record with ascii value
    seq:
      - id: index
        type: u1
      - id: name
        type: str
        size: 12
        encoding: UTF-8

  struct_b:
    seq:
      - id: some_other_elements
        type: element
        repeat: expr
        repeat-expr: 5

  element:
    doc: element with index
    seq:
      - id: index
        type: u1
    instances:
      want_to_store_ascii_here:
        value: ?

Mikhail Yakshin

@GreyCat

Can I ask you to put these in triple backticks, so it will be formatted properly?

Curtis Mechling

@cmechlin

sorry

Mikhail Yakshin

@GreyCat

Ok, so essentially it looks like you need to go up to the level of "sections",

pick a neighboring "section", and treat that "section" as known-to-be-of-specific header.type?

Mikhail Yakshin

@GreyCat

If that's the case, you can pass _index from "sections" definition as a parameter...

and use something like

  element:
    doc: element with index
    seq:
      - id: index
        type: u1
    instances:
      want_to_store_ascii_here:
        value: '_root.sections[_parent._parent._parent.index_param - 1].body.as<struct_a>.records[index].name

Curtis Mechling

@cmechlin

got it. fantastic! let me give it a try. i was able to add the array as an instance by hardcoding the index in starting from _root down, but unfortunately the structs can be in any order, and the header type field is what identifies them

Mikhail Yakshin

@GreyCat

Well, if you can't locate the other instance by calculating the index, then probably this solution won't work

Unfortunately, if you need something like "find a section in this array which satisfies this condition", then so far we don't have a good solution for that

(except for plugging in native imperative code, which you totally can try, e.g. with a custom opaque type)

Petr Pučil

@generalmimon

Unfortunately, if you need something like "find a section in this array which satisfies this condition", then so far we don't have a good solution for that

actually, that's something that plugging a value instance recursively referencing its previous self can perfectly handle; the only pitfall is that you need to tell KSC the type of that instance (by using type casting or using params, where you always have to specify the type), since it is unable to derive it because of that recursive reference (so you get a cryptic None.get in that case)

Petr Pučil

@generalmimon

@GreyCat @dgelessus I've just pushed a bunch of CI dashboard improvements - most notably implemented kaitai-io/kaitai_struct#823, but also made the table header sticky (so you can always see what language corresponds to the column that you're looking at), aggregated the individual target results in the info box (so that you can access all data that you can in the non-summarized table), made the info box a bit nicer, etc. You can look at all commits here - I tried to keep commits atomic, so you can see all individual improvements listed in the commit titles.

Please let me know what you think about it.

dgelessus

@dgelessus

Very nice, the entire width of the table fits onto my laptop screen now :)

dgelessus

@dgelessus

uh, I take that back - it doesn't actually fit, my browser still had a 70% zoom setting saved for the page :D But still much better than before, where even with zooming out I had to scroll horizontally a lot.

Mikhail Yakshin

@GreyCat

@generalmimon That's uber cool!

"mixed" striped background is my favourite ;)

Petr Pučil

@generalmimon

@dgelessus @GreyCat thanks for your feedback!

FabulousCupcake

@FabulousCupcake

has anyone ever thought about using kaitai in shell in a manner like jq? 🤔

something like… ksq <struct.ksy> <file.gif> .logicalScreen.imageHeight

I want to use kaitai but don't want to maintain a codebase for it lol

dgelessus

@dgelessus

It's probably not exactly what you're looking for, but the Web IDE can display object trees for any KSY, and can also export (sub)trees as JSON. The Web IDE is of course web-based and there's no way to access the JSON export feature from the command line or anything.

FabulousCupcake

@FabulousCupcake

yea, it's possible to do it via the web ide manually

I just thought it would be neat if there's a CLI tool to do it

makes it very scriptable in bash (or any shell)!

The Dread Vixen Alinsa

@alinsavix

That would actually be pretty neat.

Petr Pučil

@generalmimon

@FabulousCupcake see my answer at https://stackoverflow.com/a/66875706 - you can use ksdump to get the JSON object tree (see the Stack Overflow answer to see how to get ksdump) and then just use jq on it:

$ curl -L -o sample.gif https://sample-videos.com/gif/3.gif
$ ksdump -f json sample.gif image/gif.ksy | jq '.logical_screen_descriptor | { width: .screen_width, height: .screen_height }'
Compilation OK
... processing image\gif.ksy 0
...... loading gif.rb
Classes loaded OK, main class = Gif
{
  "width": 312,
  "height": 312
}

However, using ksdump might have some performance + memory impact if the .ksy spec reads some big data chunks in lazily parsed instances, which would have not been parsed unless you specifically request them, but ksdump makes sure that all such lazy instances are invoked and included in the object dump. So if your goal is just to read width/height of the image, it might be a better idea to write a little bit of code to get just these values:

$ kaitai-struct-compiler -t python image/gif.ksy
$ python3 -m pip install kaitaistruct
$ python3 -c 'from gif import Gif; g = Gif.from_file("sample.gif"); '\
'print("width: {:d}, height: {:d}".format(g.logical_screen_descriptor.screen_width, g.logical_screen_descriptor.screen_height))'
width: 312, height: 312

All seq fields (that are not in instances) will still get parsed, but at least the instances will be left uninitialized. The question is whether the KSY spec makes significant use of them or not. If not, it will not have much performance benefit compared to the ksdump + jq approach.

FabulousCupcake

@FabulousCupcake

cool, thank you!

The Dread Vixen Alinsa

@alinsavix

actually, that's something that plugging a value instance recursively referencing its previous self can perfectly handle; the only pitfall is that you need to tell KSC the type of that instance (by using type casting or using params, where you always have to specify the type), since it is unable to derive it because of that recursive reference (so you get a cryptic None.get in that case)

...I wonder if this would be applicable to my nested-arrays problem in some way.

Petr Pučil

@generalmimon

@alinsavix

...I wonder if this would be applicable to my nested-arrays problem in some way.

hm, what is your nested-arrays problem, actually? Can you put it more specifically?

The Dread Vixen Alinsa

@alinsavix

So a single array in this format looks like:

    m2array:
        seq:
            - id: num
              type: u4
            - id: offset
              type: u4
        instances:
            values:
                pos: offset
                type: u4
                repeat: expr
                repeat-expr: num

... the nested version is basically the same array, but with a datatype of 'm2array', so basically wrapping the above in:

    m2array_nested_outer:
        seq:
            - id: num
              type: u4
            - id: offset
              type: u4
        instances:
            values:
                pos: offset
                type: m2array
                repeat: expr
                repeat-expr: num

There was something wonky/strange/something about trying to get that to behave reasonably ... but now I forget what the something was. Been a month or so since I was poking at it, kind of gave up to move onto other things.

Where communities thrive

People

Repo info

Activity