Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 14:41
    J-T-de commented #422
  • 14:03
    armijnhemel synchronize #428
  • 10:39

    generalmimon on read-bits-int-le

    (compare)

  • 09:48

    generalmimon on runtime-install-instructions

    (compare)

  • 09:48

    generalmimon on master

    Add some pointers to runtime in… Use `python3 -m pip` instead of… (compare)

  • 09:48
    generalmimon closed #419
  • 09:48
    generalmimon closed #571
  • 09:47
    generalmimon commented #419
  • 03:17
    ZetaTwo commented #402
  • 03:08
    sweetgiorni commented #27
  • 03:06
    ZetaTwo synchronize #402
  • 03:04
    ZetaTwo synchronize #401
  • 00:31
    Fee0 closed #858
  • 00:13
    generalmimon edited #857
  • Mar 06 23:52
    ZetaTwo commented #172
  • Mar 06 23:36
    ZetaTwo commented #401
  • Mar 06 23:34
    ZetaTwo synchronize #401
  • Mar 06 23:32
    ZetaTwo synchronize #401
  • Mar 06 23:20
    ZetaTwo synchronize #402
  • Mar 06 22:09
    generalmimon edited #411
James Elliott
@brunchboy
I just tried again in Chrome, and get the same garbage results.
And I just noticed that in the output from your Java program, you are also getting garbage results. The last line, body.body.end_beat is impossibly high. In a correct parse, the end beat is 0x1ff, or 511. Here is a screen shot of the results using the version of the file I have in Git, which does not attempt to make unmasking conditional:
Screen Shot 2021-02-07 at 14.50.51.png
So it seems things are not working as they should for you either, which at least makes this somewhat less mysterious.
Petr Pučil
@generalmimon
@brunchboy Oh yeah, it's actually not mysterious at all - it's just that by pulling mood out of body, you completely bring the XOR mask out of sync (it's now permanently shifted by 2 bytes until the end of file).
Petr Pučil
@generalmimon
There's a simple fix: don't put raw_mood into seq, but only look at its position using an instance (+ eliminate the bit-twiddling mood instance and let mood be normally in song_structure_body).
James Elliott
@brunchboy
Ah of course!!!!
I knew I was missing something obvious. Thank you.
Petr Pučil
@generalmimon

@brunchboy

Thank you.

Glad to help :)

James Elliott
@brunchboy
Yes! I have it working with both masked and unmasked files. And the additional advantage of not moving “mood” like this means I don’t have to make changes to the other parts of the .ksy that depend on it, nor to the Java code that looks at it. :smile:
Petr Pučil
@generalmimon
@brunchboy :+1:
kariche
@KhalidAriche
Hello All,
Maven plugin problem version (0.1.4) and kaitai( 0.9) on kaitai:generate goal
Failed to execute goal name.valery1707.kaitai:kaitai-maven-plugin:0.1.4:generate (default-cli) on project **: Fail to generate Java files
[INFO] /meta/bit-endian: unknown key found, expected: application, encoding, endian, file-extension, id, imports, ks-debug, ks-opaque-types, ks-version, license, title, xref
Petr Pučil
@generalmimon

@KhalidAriche You need the Kaitai Struct compiler version 0.9 to be able to use the bit-endian key.

The kaitai-maven-plugin is apparently using the compiler bundled in the universal .zip from Bintray of version specified by the kaitai.version config option, so if you set it to 0.9, it should work.

kariche
@KhalidAriche
Thank you, it's working :thumbsup:
Petr Pučil
@generalmimon
@KhalidAriche :+1:
kariche
@KhalidAriche
how can we generate a Json object, from the generated class without _root, _parent ... (framework used fields) only fields defined in .ksy ?
Petr Pučil
@generalmimon
@KhalidAriche what target language you want to use?
kariche
@KhalidAriche
in Java
when I use Gson, but I doesn't work because of refeneces to parent, root ...
Petr Pučil
@generalmimon
@KhalidAriche you'll need to filter out the properties starting with an underscore (_)
kariche
@KhalidAriche
@generalmimon I can't avoid reflection use and filtering because it's not good for performance... ?
Petr Pučil
@generalmimon
there's a Java visualizer for Kaitai Struct at https://github.com/kaitai-io/kaitai_struct_gui, so perhaps you'd want to see how the members are accessed there?
kariche
@KhalidAriche
Ok good idea thank you 👍
Mikhail Yakshin
@GreyCat
@KhalidAriche All existing visualizers do use reflection to extract the data.
That's true for JavaScript, that's true for Ruby, that's true for Java-based ones.
There was a tentative attempt to draft a solution that will generate toJsonString() methods everywhere that will generate JSON string for current class and all its members
but none of these were really solidified and implemented
kariche
@KhalidAriche
how to specify the --no-auto-read on kaitai compiler maven plugin ?
Petr Pučil
@generalmimon

@KhalidAriche

how to specify the --no-auto-read on kaitai compiler maven plugin ?

I assume that the https://github.com/valery1707/kaitai-maven-plugin does not support that, there is nothing like that in the documented plugin parameters or anywhere in the code (if I used the right needles).

You can contact @valery1707 if he doesn't want to add support for it, open an issue on that repo or even a pull request with the implementation (it can't be anything hard; register a new plugin parameter noAutoRead, then look at its value when invoking kaitai-struct-compiler and pass the --no-auto-read CLI arg if it's true; 6 added lines at most). Or you can resort to messing up with the command-line ksc directly without the Maven plugin intermediary - then you will have the most control.

Also note that the kaitai-maven-plugin is not managed by the Kaitai team - it even doesn't look like anybody else than @valery1707 actually has write access into that repo. It's not so important (we'll keep doing our best to help you anyway), but I'm just saying that we don't have any more control over that plugin than you do.

James Elliott
@brunchboy
Thanks for the clarification. I agree with all that (and already understood it) but want to emphasize that the plugins are a really important part of the Kaitai ecosystem. They make it practical to use in a Maven or Gradle based development and continuous integration environment, without any need to learn how install and configure and invoke the compiler. It also means people can work on my project which embeds Kaitai (via the Maven plugin) without needing to learn or do that stuff. Without the plugin, I doubt I would have adopted Kaitai in the first place, and I am very glad I did. If the original developer isn’t interested in further enhancements, hopefully he’ll allow and encourage someone else to take over.
Tim Kurvers
@timkurvers

hi folks! does Kaitai support defining composite types of arbitrary Kaitai struct types? for example, given CustomArray<CustomType> entries in C++, I naively tried the following in Kaitai:

- id: entries
  type: custom_array(custom_type)

but that does not work as it understandably expects custom_type to be a field, rather than a Kaitai type. thoughts?

Tim Kurvers
@timkurvers
might require template/generics suggested in: kaitai-io/kaitai_struct#135
waffle-cop
@waffle-cop

Is there any way to set up a type switch within a loop, being switched on the previous element in the loop?
I Tried this but was not having any luck due to this error "don't know how to call method 'name' of object type 'KaitaiStructType'"

types:
  fields:
    seq:
      - id: fields
      type: 
        switch-on: _.name
        cases:
          "XXXX": unknown_field_known_size(_.value)
          _: unknown_field
      repeat: until
      repeat-until: _._io.pos == _parent.size

Basically I am looping over a set of key/value pairs that follow the same format, the only outlier being that if the size of the value would require more than 2 bytes to store, the size is then stored in the previous pairs value, signified by the key of the previous value being "XXXX".

Is there another way to go about this that I missed?

Mikhail Yakshin
@GreyCat
@waffle-cop You can always encapsulate that into yet another type, possible parametric one
Matthew Turk
@matthewturk
Do folks think it would be feasible to have a typescript interface to the compiler? I'm new to scala but it looks like generating the interface files or generating ts directly are both available as sbt plugins.
Matthew Turk
@matthewturk
Oh! It looks like the webide bundles some definition files. https://github.com/kaitai-io/kaitai_struct_webide/blob/master/lib/ts-types/kaitai.d.ts
Georg Friedrich
@Gansgar
Hey. I'm really fond of the kaitai system. It looks great to reverse engineer, document and implement some protocol definitions. But is it possible to use the web ide with local filesystem storage? I've seen the serve_files.py file in the git, but there's no update since 2017 on it and no documentation anywhere
Matthew Turk
@matthewturk
As a note I've been experimenting with a jupyterlab kaitai extension. Right now, it just does HexViewer from the webide. This is in compliance with the license, although if it should either be changed, have more credits I should add to the readme, or if folks would rather I not do this let me know and I'm happy to comply. https://github.com/matthewturk/jupyterlab_kaitai It's pretty rough around the edges so far -- not nearly as nice as the webide. :)
Petr Pučil
@generalmimon

Hey. I'm really fond of the kaitai system. It looks great to reverse engineer, document and implement some protocol definitions. But is it possible to use the web ide with local filesystem storage? I've seen the serve_files.py file in the git, but there's no update since 2017 on it and no documentation anywhere

Hi @Gansgar, as I can see in the commit that added serve_files.py, using local filesystem is a feature exclusive to version v2 of the Web IDE. The currently deployed version is v1. @koczkatamas was actively working on v2 about 4 years ago (in 2017), but nowadays, it's just an unfinished abandoned subproject (though a lot of work has been already done - see kaitai-io/kaitai_struct_webide#35). Hopefully someone will return to it and finish the work.

The current state of v2 is available at https://ide.kaitai.io/devel/v2.html, but it looks like there are some errors which prevent from using certain features (to the extent that I doubt whether it's even usable at this time). Although all file/folder operations seem to work fine, I haven't even figured out how to compile a .ksy spec (every time an error occurs) - maybe all that needs to be done is a 5-minute fix, maybe not. Also I'm not sure how to activate the local filesystem feature - apparently the serve_files.py script only allows requests that authenticate themselves with a 16-byte hex secret, which has to be passed in a query parameter secret to the Web IDE. Then you'd need to add a node for the "remote file system" to the directory tree (more specifically, to this list) to be able to access it with the UI, and just fix any errors/bugs as they come :)

So yeah, I'd say it is possible, just not without any work :smile: It's an unexplored area, so expect various pitfalls and bugs. However, if you feel brave enough to give it a try, I'd be happy to accept a PR with necessary changes and fixes, and possibly replace the v1 version with v2 once it's finished. Although it is not yet finished, I can see that v2 comes with a plenty of new features and improvements, so it would be great to get it working.

Unfortunately, it is not very high on my priority list right now - you might ask @luxaritas as he seemed interested in the exact same issue 5 months ago (see https://gitter.im/kaitai_struct/Lobby?at=5f6180d74f19dd093350c994). He wrote that he would be able to take a peek in the next couple weeks (written at September 17, 2020 in https://gitter.im/kaitai_struct/Lobby?at=5f63b81bce5bbc7ffdcfe553), so perhaps he already achieved some success.

Spencer Goh
@dymaxionuk
Hi. I really like the concept of Kaitai Struct, but for pcap files it doesn't work at the moment. There's a magicnumber per packet header rather than per file which indicates the endianess of the packet.This number however has 2 forms which signifies whether or not the packet has microsecond or nanosecond timestamps. Currently the ksv file on website supports a fixed endianess le/be, but suspect doesn't allow per packet endianess (since it would require a meta endian switch section with self reference to the header (rather than _root that I've seen in examples)
Also there would also be required a switch on the magic number to determined the packet timestamp format.
So overall i really like this, and it's much faster 2x vs Pcap4j, however it simply doesn't work, if i bodge the generated code to be trivially lenient of both magicnumber formats, the code continues blindly but eventually blows up with some parse errors. I haven't looked enough into the code to figure out why, but given time constraints I'm having to abandon this library for pcap4j simply sure to parse errors. Is the pcap parsing fixable given current constraints of Kaitai struct APIs ?
John Chadwick
@jchv
Should be possible to switch endian on a packet level. AFAIK you can put the endian meta in any struct and it can be calculated in any location.
The pcap ksy is old though and I am guessing it may need to be fixed still. Before 0.8 this was not possible.
Spencer Goh
@dymaxionuk
What about the timestamp, magic number packet based, micro/nanosecond switch, do you think that would be supported by the current code?
If I'm theory can, then i might look into it, but given my current workload i don't want to investigate if it's literally impossible, by i don't know enough about the internals of this library to decide whether to investigate.
Thanks in advance
Petr Pučil
@generalmimon

Hi @dymaxionuk.

What about the timestamp, magic number packet based, micro/nanosecond switch, do you think that would be supported by the current code?

Yes, absolutely.

but for pcap files it doesn't work at the moment. There's a magicnumber per packet header rather than per file which indicates the endianess of the packet. This number however has 2 forms which signifies whether or not the packet has microsecond or nanosecond timestamps.

I'm not sure if we mean the same format, but the specification at https://wiki.wireshark.org/Development/LibpcapFileFormat says otherwise. The magic_number seems to be really only present in the Global Header, which is present only once at the beginning of the pcap file:

Global Header

This header starts the libpcap file and will be followed by the first packet header:

I can't see any other magic_number in the Record (Packet) Header.

Although they suggest that the wiki at wiki.wireshark.org is deprecated and the one correct is https://gitlab.com/wireshark/wireshark/-/wikis/Development/LibpcapFileFormat#global-header, I can't spot any difference in terms of information provided by the new page. It only treats the pcap format with nanosecond precision as a variant of pcap (not the core pcap itself, as the old wiki did), which is called Nanosecond pcap, but the conclusion is the same.

So if my interpretation is correct, I'd say the simplest solution is the following:

meta:
  # ...
  # "endian: le" should be removed from here because the spec no longer uses one predominant endianness
seq:
  - id: magic_number
    type: u4be
    enum: magic_numbers
  - id: body
    type: pcap_body
instances:
  has_nanosec_resolution: # this will be probably useful as well - you can
                          # access it from anywhere as `_root.has_nanosec_resolution`
    value: |
      magic_number == magic_numbers::nanosec_le
      or magic_number == magic_numbers::nanosec_be
types:
  pcap_body:
    meta:
      endian:
        switch-on: _root.magic_number
        cases:
          magic_numbers::microsec_le: le
          magic_numbers::nanosec_le: le
          magic_numbers::microsec_be: be
          magic_numbers::nanosec_be: be
    seq:
      - id: hdr
        type: header
      - id: packets
        type: packet
        repeat: eos
  header:
    doc-ref: 'https://wiki.wireshark.org/Development/LibpcapFileFormat#Global_Header'
    seq:
      # note: extracted `magic_number` from here to root `seq`
      - id: version_major
        type: u2
      - id: version_minor
        type: u2
enums:
  magic_numbers:
    0xa1b2_c3d4: microsec_be
    0xd4c3_b2a1: microsec_le
    0xa1b2_3c4d: nanosec_be
    0x4d3c_b2a1: nanosec_le

Note that if the parsed value of magic_number does not match anything in /types/pcap_body/meta/endian/cases, a KaitaiStream.UndecidedEndiannessError is thrown, so it's guaranteed that an invalid file is rejected as soon as possible.

However, if you are convinced that you need "per packet endianness" switch instead of the per file as it is now, it is still perfectly possible. There is no technical limitation telling that you can't do that: meta/endian/switch-on can be put on any subtype, and you can reference the parent struct by using _parent just as well you can access the top-level struct with _root. Moreover, if "pulling" the value of a field from an outer type using _root.{...} or _parent.{...} doesn't suit you or you need more flexibility, you can always "push" the values you need "from outside to the inside" using parameters.