Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 06:20
    KOLANICH commented #906
  • 00:44
    jmerdich opened #906
  • Sep 21 00:01
    KOLANICH opened #905
  • Sep 19 18:41
    KOLANICH commented #889
  • Sep 19 18:39
    KOLANICH commented #889
  • Sep 19 18:39
    KOLANICH commented #889
  • Sep 19 18:38
    KOLANICH commented #889
  • Sep 18 19:31
    generalmimon synchronize #469
  • Sep 18 15:21
    k2on commented #10
  • Sep 15 14:13
    armijnhemel opened #534
  • Sep 15 14:06
    gargan26 commented #889
  • Sep 15 09:19
    N-Nagorny commented #509
  • Sep 15 09:17
    N-Nagorny synchronize #509
  • Sep 15 07:41
    PeterWendt opened #45
  • Sep 15 06:32
    PeterWendt commented #44
  • Sep 14 18:52
    KOLANICH synchronize #438
  • Sep 14 18:52
    KOLANICH synchronize #442
  • Sep 14 18:52
    KOLANICH synchronize #315
Mikhail Yakshin
@GreyCat
Ok, so since the discussion in kaitai-io/kaitai_struct#136, I believe we no longer package formats library into the same .deb package as the compiler itself — kaitai-struct-compiler*.deb
Mikhail Yakshin
@GreyCat
The second file we'll be looking for is kaitai-struct-formats*.deb, and it looks like we haven't moved this one from previous distribution site to GitHub releases...
Petr Pučil
@generalmimon

The second file we'll be looking for is kaitai-struct-formats*.deb, and it looks like we haven't moved this one from previous distribution site to GitHub releases...

I don't think it ever existed in the first place - it's proposed in kaitai-io/kaitai_struct#136 but AFAIK it has never been put into practice. I can check my Bintray backup tonight, but I don't recall seeing anything like that.

I believe we no longer package formats library into the same .deb package as the compiler itself — kaitai-struct-compiler*.deb

We never did that either, to the best of my knowledge.

The only distribution with the embedded format library is the Universal .zip since version 0.7.

Rob Nelson
@N3X15_gitlab
Bit of an interesting problem. I'm working on a Kaitai of the ECMA 335 (.NET) PE extensions. Stream are defined as offset from the metadata section, so I made some code to inject variables into the generated structure that call pos() so each structure knows where it begins. Unfortunately, KSC does not like me referencing these variables. Is there a way to tell it to ignore such errors or should I just give up and patch KSC?
Mikhail Yakshin
@GreyCat
@generalmimon Ugh :( It looks like I've messed up this one...
@N3X15_gitlab Sorry, I don't quite get the problem. Which language you're targeting? And may be you can give a YAML example of what's not working?
Rob Nelson
@N3X15_gitlab
Python, and one sec.
io_start_pos is injected after generation
Mikhail Yakshin
@GreyCat
Ok, by "injected" you mean either changing it in generated .py file after generation, or patching a class in runtime?
Rob Nelson
@N3X15_gitlab
In the generated .py file.
Mikhail Yakshin
@GreyCat
Ok, so effectively ksc has no knowledge that an attribute like "io_start_pos" exists then?
Rob Nelson
@N3X15_gitlab
Correct.
Mikhail Yakshin
@GreyCat
And it doesn't know which type that would be, etc.
Currently we don't have such extension mechanism.
Rob Nelson
@N3X15_gitlab
Yeah, I'm wondering if there's a workaround, though.
Mikhail Yakshin
@GreyCat
What we have are either opaque classes — which are, well, by definition, opaque — meaning that KS still doesn't know anything about its inner world,
and we have custom processing routines
which allow you to inject some code but they're not types per se
If you're looking into getting start position of a specific member, there is a hackaround
Rob Nelson
@N3X15_gitlab
The problem is that the position has to be read at a particular time so that my code knows where the beginning of a particular section is.
Mikhail Yakshin
@GreyCat
relying on the way the parsing is done
Rob Nelson
@N3X15_gitlab
With instances, there's no guarantee
Mikhail Yakshin
@GreyCat
Right... So what you can do is something close to:
seq:
  - id: foo
    size: x
  - id: bar
    size: y
    if: ofs_bar >= 0 # always true, but calls ofs_bar right before parsing bar
instances:
  ofs_bar:
    value: _io.pos
This way you'll get ofs_bar populated with _io.pos right at the moment before bar was read
Rob Nelson
@N3X15_gitlab
Okay, I'll give that a shot. Thanks.
dgelessus
@dgelessus
Niklas
@nik-sm

Hi there - I have a bit of a newbie question. I'm working with a filetype produced by a scientific instrument.

The data is something like this:
(Magic, ... length) (body1 ... ) (body2 ...)

The body packets (in order of appearance in the file) have 60 bits of contents, and then a 4 bit flag, describing what type of body packet it is.

So I need to grab that 4 bit flag first, and then parse the remaining 60 bits accordingly.

The approach I'm currently trying is to treat this 64 bit body packet as little-endian, process the first 4 bits, and then set the type of remaining part with a switch statement.
Below is how I am currently trying to do this, but it doesn't seem to work yet.
Basically - if I specify that the body packets is just 'u8' with 'meta/endian: le' - then it definitely uses little endian.
But if I specify that it has "size: 8" and then use a custom type (with an initial "b4" field describing which body packet, and another field using that switch statement to handle the rest)

Here's what I've got so far

meta:
  id: tpx3
  file-extension: tpx3
  endian: be
seq:
  - id: tpx3_magic
    contents: 'TPX3'

  - id: chip_number
    type: u1

  - id: unused_byte
    type: u1

  - id: body_bit_length
    type: body_bit_length

  - id: num_body_packets
    repeat: expr
    repeat-expr: body_bit_length.bit_length / 8 
    type: body_packet_wrapper

types:
  body_bit_length:
    meta:
      endian: le
    seq:
      - id: bit_length
        type: u2
  body_packet_wrapper:  # An 8 byte wrapper, so we can specify little-endian
    meta:
      endian: le
    seq:
      - id: contents
        type: u8   # Using this, it correctly switches byte order, so I can see the desired byte describing "which body packet will it be?"

        # type: custom  # <-- What I want to do is specify that these 8 bytes should be little-endian, and handle the rest with more types below
        # size: 8

  custom:
    seq:
      - id: first
        type: b4

      - id: rest
        type: 
            switch-on: ...

Here's some example lines, where we see (header) (body type 1) (header) (body type 1)

00000000  54 50 58 33 00 00 08 00  20 20 b9 a6 1b 96 3a 6f  |TPX3....  ....:o|
00000010  54 50 58 33 00 00 08 00  e0 0e 90 b0 1b a6 3a 6f  |TPX3..........:o|
00000020
Niklas
@nik-sm
To clarify - In the example lines above, you can see the "6F" byte at the end of the first line. I'd like to inspect that "6F" byte first - and the "6" will indicate that it is "body type 1"
Petr Pučil
@generalmimon

@nik-sm

You can pull this off like this:

meta:
  id: tpx3
  file-extension: tpx3
  endian: be
seq:
# ...
  - id: body_packets # ~num_body_packets~
    size: 8
    type: body_packet_wrapper
    repeat: expr
    repeat-expr: body_bit_length.bit_length / 8

  body_packet_wrapper:  # An 8 byte wrapper, so we can specify little-endian
    meta:
      bit-endian: le
    seq:
      - id: rest # this will have bit size 60
        type: 
          switch-on: first
          cases:
            6: body_type_1
            # ...
      - id: first_seq # not necessary, you can remove this, just showing the position of `first`
        type: b4
    instances:
      last_byte:
        pos: 7
        type: u1
      first:
        value: (last_byte & 0xf0) >> 4

The "body type" (or the first field) is parsed using a parse instance. However, using bit-sized types in instances is not really supported (it would shift the internal bit position pointer of the IO stream without bringing it back), so a workaround is to parse the whole last_byte and extract the high nibble from it with bit masking.

I also replaced endian: le with bit-endian: le - endian: le only affects byte-level parsing, which is not present, so it's no longer necessary. bit-endian: le means the little-endian bit layout for bit-sized types (type: bX) - I assume this is the case because you said that the last 4 bits specify the body type. That fits the little-endian direction of parsing.

Now, body_type_1 is expected to contain any number of bit-sized types (type: bX), which add up to 60 bits. For example:

types:
  body_type_1:
    seq:
      - id: foo
        type: b56
      - id: bar
        type: b4

This would parse the stream from above like this:

  e0 0e 90 b0 1b a6 3a 6f
  \____ 56 bits _____/  |
                        4 bits

  e0   0e   90   b0   1b   a6   3a   6f
 │<─┃ │<─│ │<─│                
 ╷    ╷  ↑ ╷  ↑
 └┈┈┈>┈┈┈┘ ┊  ┊    ⋮
      └┈┈┈>┈┈┈┘    ┊
           └┈┈┈>┈┈┈┘

               …│<─│<─│<─┃
  foo: 0x      …_90_0e_e0

    6   f
    hi  lo
       │<─│
       `bar`

  bar: 0xf
( first_seq: 0x6 )

I hope it's clear. See https://doc.kaitai.io/user_guide.html#bit-ints-le for verbal explanation and other diagrams.

1 reply
Joseph Oravec
@joeyoravec

Can anybody think of a trick to get a member variable into the kaitai parser besides global variables or modifying the generated code? My initial thought was to feed something into the constructor so it has the instance-specific context:

std::istringstream is(buf);
kaitai::kstream ks(&is);
example_t data(&ks, MyExtraVariable);

but even if it got persisted to a member variable, it won't be available to a process routine or an opaque type constructor because there's no pointer back to the kstruct. I couldn't figure out any other way with base class / derived class. Any tricks you can think of? For now I'm just coupling my process/opaque with global variables for instance specific data

4 replies
cmhulbert
@cmhulbert_gitlab

Hey eveyone, I'm getting this error in the devel web ide when trying to parse some test data:

Parse error (TypeError): Cannot read property 'indexOf' of null
Call stack: TypeError: Cannot read property 'indexOf' of null
    at Parser.cleanup (https://ide.kaitai.io/devel/lib/_npm/yamljs/yaml.js:1220:15)
    at Parser.parse (https://ide.kaitai.io/devel/lib/_npm/yamljs/yaml.js:754:23)
    at Function.Yaml.parse (https://ide.kaitai.io/devel/lib/_npm/yamljs/yaml.js:1817:25)
    at JsImporter.importYaml (https://ide.kaitai.io/devel/js/v1/KaitaiServices.js:51:33) TypeError: Cannot read property 'indexOf' of null

I haven't used Kaitai in a while, but a few months ago when I last used these same ksy files and test data, I didn't have this issue. Is there maybe some simple thing I'm doing wrong?

18 replies
Brian Manlove
@N129BZ
Any possibility of better documentation (ANY documentation :) of using kaitai with go? Ksc will generate a go struct but there are no examples of actually using it...
Petr Pučil
@generalmimon

@N129BZ All Kaitai Struct generated parsers rely on a runtime library for the particular language, for Go there is https://github.com/kaitai-io/kaitai_struct_go_runtime. In a Go project with a go.mod file (initialized by go mod init), run

go get github.com/kaitai-io/kaitai_struct_go_runtime/kaitai@0.9

You should use @0.9 if you use the stable KS compiler (kaitai-struct-compiler 0.9), because otherwise you'd get the latest development version of the Go runtime that may not work with the KSC 0.9. Right now, there are no such breaking changes, but there may be some in the future.

(If you use Go 1.17, go get is deprecated, so you may use a different command - but the path will be the same.)

Once you have the .ksy file compiled to a Go parser that uses the same package name as your application code (use --go-package <name> command line option of kaitai-struct-compiler, e.g. kaitai-struct-compiler -t go --go-package my_app gif.ksy for a package my_app declaration), import the Go runtime library (that you installed previously) and use the parser in your application as follows (from https://kaitai.io/#quick-start):

package my_app

import (
    "github.com/kaitai-io/kaitai_struct_go_runtime/kaitai"
    "fmt"
)

// ...

file, err := os.Open("path/to/some.gif")
// if err != nil { ... }
g := NewGif()
err = g.Read(kaitai.NewStream(file), g, g)
// if err != nil { ... }

fmt.Printf("width = %d\n", g.LogicalScreen.ImageWidth)
fmt.Printf("height = %d\n", g.LogicalScreen.ImageHeight)
Brian Manlove
@N129BZ
Thank you, I had not seen any "go" option in the sample codes listed. HOWEVER, I am now getting the following error when I debug: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (ST1006)go-staticcheck. That error is happening because of this code generated by ksc: func (this AhrsRecord) Read(io kaitai.Stream, parent interface{}, root *AhrsRecord) (err error) My application is reading UDP packets and I have successfully used kaitai-struct and kaitai-stream to parse my UDP data in csharp, javascript, and python - but not go. This is the runtime error /program.go:13:7: undefined: NewAhrsRecord (exit status 2)
func NewAhrsRecord() *AhrsRecord {
    return &AhrsRecord{}
}

func (this *AhrsRecord) Read(io *kaitai.Stream, parent interface{}, root *AhrsRecord) (err error)
alanjtaylor
@alanjtaylor

I am currently using kaitai 0.9 to to parse an array of events.

When I try to run my code, I get the error:

stack smashing detected : terminated

I have a feeling that it could be that one of the events in the array doesn't match the schema. Is there anyway to demand that kaitai checks that it doesn't overflow some buffers?

3 replies
Brian Manlove
@N129BZ
Finally got my kaitai-struct/go project to run. Not so much a kaitai problem as it was/is my inexperience with getting go set up correctly. That being said, why does ksc generate func code with "parent" and "root" method parameters? I would think stuff like that should be generated via the structure of the ksy file. I edited that out for my implementation, so the end result of my previously generated go file is much simpler...
func NewAhrsRecord() *AhrsRecord {return &AhrsRecord { }
}

##      original generated function:
#  func (this *AhrsRecord) Read(io *kaitai.Stream, parent interface{}, root *AhrsRecord) (err error) {
#       # do stuff here
#  }

## after editing for simplicity:
func (*AhrsRecord) Read(io *kaitai.Stream, ahrsrecord *AhrsRecord) (err error) {
     # do stuff here
}
Brian Manlove
@N129BZ
Wouldn't parents and roots be elements of a linked list or something like that? I just want to decode a simple UDP message payload of a byte array into a struct with it's properties... ;-)
James Elliott
@brunchboy
In .ksy files you can refer to elements that come from the parent of the current node, or even starting from the root, so in order to correctly read a structure specified by such a file, the reader needs access to the parent and root pointers. At least in the Java versions, there are three different constructors, one in which you pass all three elements like in your example above, one in which you just pass the stream and parent (which just calls the three-element version with a null root), and one in which you only pass the stream, which calls the three-element version with null values for both root and parent. Doesn’t the Go code have those three variants?
6 replies
Brian Manlove
@N129BZ
@brunchboy the compiler is also adding 3 elements to the struct that are completely unnecesary. I don't need the returned struct to contain references to the stream, or a parent, or a root... I just want the fields I defined in the .ksy file... internally it's assgning the passed in object references to a "private" this._io, this.parent, this._root, I can just use the passed in references directly without the additional baggage of yet more internal stuff... and to be redundant, GO complains about using "this"
type AhrsRecord struct {
    Btid uint8
    Msgid uint8

    etc...

       // why do I need this stuff?   If I wanted it I would have indicated that in the ksy file
    _io *kaitai.Stream
    _root *AhrsRecord
    _parent interface{}
}
James Elliott
@brunchboy
My understanding is that the structs are created to work well with the reading process, and those fields are necessary for that. If you want your own structs that don’t have those fields, and are not used as part of the reading process, you can define them manually, and copy the information that Kaitai parses from its own structs into yours. The internals of the generated structs are not supposed to be pretty, but they are easy to create from the .ksy spec, and easy to fill with information from files you want to parse.
alexander-minchin
@alexander-minchin

Hi all,
would someone be able to point me to the best practice for extracting all of the decoded items out of the kaitai struct object and into a dictionary (in Python)?

What i'm doing now is a bit messy, i'm basically calling kaitai_object.__dict__ and then filtering the result to remove any item where the key starts with an underscore :/ (ie _parent, _root, etc) .

James Elliott
@brunchboy
I just don’t think Kaitai is intended to do the things you and Brian are hoping it will. It is not for generating source code for you to use or care about in fine detail, it is generating source code for itself to use in parsing arbitrary binary structures based on your specifications (and keep in mind that ksy files can be used by other ksy files as part of parsing hierarchical binary structures, where children are allowed to reference elements from parents, so it may always need the parent and root pointers in each structure). Kaitai will take you a certain distance, using code it generates for its own purposes, and if you want to go beyond that, you write your own code to interact with the values it has parsed using its code. But I will let the developers weigh in more definitively when they have time.
Brian Manlove
@N129BZ
Whether intended or not, it does exactly what I wanted and needed it to do... it completely abstracts away the messy details of decoding bytes into useable values... don't get me wrong, I'm not trying to make kaitai do something it wasn't intended to do, I just need to understand why it does what it does. Also, as far as go is concerned, I think there's got to be a way to generate code that doesn't make the compiler issue "bad form" warnings ;-)
James Elliott
@brunchboy
That’s great, and I am sure you are right about the warnings. I don’t know who, if anyone, is working on the Go port, but they would probably love a PR to fix them. And hopefully the developers will be able to weigh in with more authoritative information than my speculations as a happy user. :smile:
1 reply
hmx12-multi
@hmx12-multi
image.png
hi! dunno if i'll get an answer, but i figured it was worth a shot to ask. i'm pretty new to this and i'm trying to use kaitai struct to grab text from some game files. it's been pretty straightforward except for the fact that i don't know how to make repetition stop when it sees the word 'root' in the file, because there's no set number of repeats and that's the only thing that designates the change of section. maybe there's a better way to do this? i'm not sure, but any help would be greatly appreciated.
dgelessus
@dgelessus
@hmx12-multi Hi! (sorry for the late reply)
I think you're looking for repeat: until, which lets you control the repetition using a condition instead of a number. See https://doc.kaitai.io/user_guide.html#_repeat_until_condition_is_met
You probably also need to use the trick described in kaitai-io/kaitai_struct#445, because your repeat end marker doesn't have the same format as the repeated items.
Andy Jarosz
@MadlyFX
Hey all, I've build a definition with a switch case to specify it's type. When using c++, what's the proper way to read that value? I.E. do I need to cast a specific type when passing into a function etc
2 replies