kaitai_struct/Lobby

dgelessus

@dgelessus

b1 parses a single bit - u1 is a single byte (unsigned)

Xniveres

@Xniveres

ah yea

Mikael Falkvidd

@mfalkvidd

Hi.
I'm parsing data from a water meter. The real value is multiplied by 1000 and then value is sent as u4. A typical value might be 10.272 (sent as 10272).
I'm currently doing this:

seq:
  - id: volume_multiplied_by_1000
    type: u4
instances:
  volume:
    value: volume_multiplied_by_1000 / 1000.0

But I would prefer to only show the final volume value in the result (not volume_multiplied_by_1000). Is there a better way I could describe the data?

1 reply

Christophe Creeten

@ccreeten

Hi guys, I just started out trying Kaitai (it's neat!) and have some questions about it. Note that I am using it in Java:

what is the status of integer expressions being done in 32-bit (CalcIntType), all the casts to 64-bit are a bit ugly (and forgetting such a cast error prone)
are there any ideas for referencing custom target language implemented types? for example, a bit like opaque types are now, but we could specify the result type in the DSL so it can be referenced and used
is it possible to describe for example a block structure, where a certain type field determines the existing following fields, in such a way that the resulting structure is condensed (all the properties of a certain block are at the same level/in the same object), e.g.:
```
block:
seq:
- id: index
  type: u1
- id: type
  type: u1
# or switch
- id: block_0_properties
  type: block_0_properties
  if: type == 0
- id: block_1_properties
  type: block_1_properties
  if: type == 1
```
Which results in:
```
Block0 {
index();
type();
// and also block_0 properties
}
Block1 {
index();
type();
// and also block_1 properties
}
```
And without making the description too complex? (I guess I'd ideally just want to say: now follows either block_0 or block_1). Perhaps even flatten multiple levels deep?

Mikael Falkvidd

@mfalkvidd

I'm getting Call stack: undefined Error: readBitsIntLe: the maximum supported bit length is 32 (tried to read 48 bits) when trying to read a 48-bit value. Is there a reason Kaitai limits the size to 32 bits? I also tried u6 but that isn't supported either.

Petr Pučil

@generalmimon

@mfalkvidd

Is there a reason Kaitai limits the size to 32 bits?

Only JavaScript (the target language that Web IDE runs) has this limitation. It's because we use JavaScript's bitwise operators in the runtime library methods used for parsing the bit-sized integer, and JavaScript has such design choice that... (MDN / Bitwise operators):

The operands of all bitwise operators are converted to signed 32-bit integers in two's complement format, except for zero-fill right shift which results in an unsigned 32-bit integer.

So if there is an attempt to read more than 32 bits, the current implementation knows it can't do that without necessarily discarding bits and returning an incorrect value, so it deliberately throws an error.

However, the methods are really supposed to work for any b<=32 integer even in JavaScript, but recently @kbudka has found (kaitai-io/kaitai_struct_javascript_runtime#20) that this doesn't work if for example the b32 integer starts on an unaligned bit position. This is something we'll definitely fix.

1 reply

Marek Sebera

@marek.sebera:matrix.org

[m]

hey, is there any way I could convert BNF to Kaitai definitions?

(except for manually ofc)

Andreas

@andreas-volz

Hi, I tried to read a 8bit and a 16bit big 1bit field from a file. I tried several variants from the docu, but I failed until now. I thought this "Bit-sized integers" feature might help. But it didn't work as expected. Here is the old C++ code I need to port:
'''
Form1->UnitSEGroupNeutral->Checked = bool((0x80&unit.Data[EntryIDunit.offcount+45])>>7);
Form1->UnitSEGroupInd->Checked = bool((0x40&unit.Data[EntryIDunit.offcount+45])>>6);
Form1->UnitSEGroupFactory->Checked = bool((0x20&unit.Data[EntryIDunit.offcount+45])>>5);
Form1->UnitSEGroupBuilding->Checked = bool((0x10&unit.Data[EntryIDunit.offcount+45])>>4);
Form1->UnitSEGroupMen->Checked = bool((0x8&unit.Data[EntryIDunit.offcount+45])>>3);
Form1->UnitSE0x8000->Checked = ((0x8000&unit.Data[EntryIDunit.offcount+54])>>15);
Form1->UnitSE0x4000->Checked = ((0x4000&unit.Data[EntryIDunit.offcount+54])>>14);
Form1->UnitSE0x2000->Checked = ((0x2000&unit.Data[EntryIDunit.offcount+54])>>13);
Form1->UnitSE0x1000->Checked = ((0x1000&unit.Data[EntryIDunit.offcount+54])>>12);
Form1->UnitSE0x800->Checked = ((0x800&unit.Data[EntryIDunit.offcount+54])>>11);
Form1->UnitSE0x400->Checked = ((0x400&unit.Data[EntryIDunit.offcount+54])>>10);
Form1->UnitSEBW->Checked = ((0x200&unit.Data[EntryIDunit.offcount+54])>>9);
Form1->UnitSELoc->Checked = ((0x100&unit.Data[EntryIDunit.offcount+54])>>8);
Form1->UnitSEUnitHero->Checked = ((0x80&unit.Data[EntryIDunit.offcount+54])>>7);
Form1->UnitSENonLoc->Checked = ((0x40&unit.Data[EntryIDunit.offcount+54])>>6);
Form1->UnitSEDoodadState->Checked = ((0x20&unit.Data[EntryIDunit.offcount+54])>>5);
Form1->UnitSEAllRaces->Checked = ((0x10&unit.Data[EntryIDunit.offcount+54])>>4);
Form1->UnitSEPlayerSet->Checked = ((0x8&unit.Data[EntryIDunit.offcount+54])>>3);
Form1->UnitSEBriefing->Checked = ((0x4&unit.Data[EntryIDunit.offcount+54])>>2);
Form1->UnitSEListing->Checked = ((0x2&unit.Data[EntryIDunit.offcount+54])>>1);
Form1->UnitSENonNeutral->Checked = (0x1&unit.Data[EntryID*unit.offcount+54]);
'''
I tried also example from website:
'''
seq:

id: packed_1
type: u1
instances:
version:
value: (staredit_group_flags & 0b11110000) >> 4
len_header:
value: staredit_group_flags & 0b00001111
'''
and I get from the kaitai compiler:
Parse error: undefined
Call stack: undefined io.kaitai.struct.precompile.ErrorInInput: (main): /instances/version/value: can't apply operator BitAnd to CalcArrayType(Int1Type(false)) and Int1Type(false)
'''
What am I doing wrong? (And why is Gitter so hard to format? :-( )

2 replies

oldbug

@oldbug:matrix.org

[m]

Just use md syntax to format your text

Andreas

@andreas-volz

Hm, I found out that there're several versions of the binary file I'm parsing are available. All formats are very similar, but I sometimes in the middle of the file a data block is removed or added. This is hard to handle as the original engineers decided to not add of version stamp I could use to detect that.
Currently while development I do a if:1=2 or if:1=1 to enable or disable some data blocks. Just to work on that three different version that I have meanwhile.
But what do you suggest is a good strategy to deal with such binaries? I could collect&maintain all MD5 sums from binaries that I identify over time and then choose for parser strategy. But that would need a if: variable that I could control from outside of the Kaitai parser.
Any hints to that special problem?

Petr Pučil

@generalmimon

@andreas-volz

But that would need a if: variable that I could control from outside of the Kaitai parser.

You can make the top-level type parametric and control the version with this parameter:

meta:
  # id: ...
params:
  - id: version
    type: u1
seq:
  # ...
  - id: some_section_only_in_v3
    type: whatever
    if: version == 3

If you use the Web IDE, having a top-level parameter won't work since there is no built-in way to pass a value to it (in theory, you could make some changes to the generated code in the JS code (debug) tab and then press Ctrl+Enter to reparse the file with the modified code every time, but that's cumbersome and annoying). In that case, I often temporarily comment the parameters and create instances with same names (https://doc.kaitai.io/user_guide.html#_value_instances), whose values can be comfortably changed. For example:

meta:
  # id: ...
# params:
#   - id: version
#     type: u1
seq:
  # ...
instances:
  version:
    value: 3

Obviously, when you're done with the manual testing in the Web IDE, you uncomment params and delete the temporary instances - then the spec is ready for application use.

2 replies

Mikael Falkvidd

@mfalkvidd

Is there a way to make the Kaitai javascript generator set enumerable to true on instances? It seems like normal properties are enumerable, but not instances.
Alternatively, is there a way to iterate over all properties (enumerable and non-enumerable) in ES5? I have tried Object.getOwnPropertyNames() but that doesn't include the instance proerties, regardless if they are enumerable or not. Manually changing them to enumerable works (when using for...in)

Petr Pučil

@generalmimon

@mfalkvidd

Is there a way to make the Kaitai javascript generator set enumerable to true on instances? It seems like normal properties are enumerable, but not instances.

Makes sense, thanks for the suggestion! We're using the Object.defineProperty to define the instance, but I didn't realize that it creates a non-enumerable property by default:

enumerable
true if and only if this property shows up during enumeration of the properties on the corresponding object. Defaults to false.

1 reply

Alternatively, is there a way to iterate over all properties (enumerable and non-enumerable) in ES5?

See what I did in my code to dump all properties of a Kaitai Struct object in JavaScript.

6 replies

Petr Pučil

@generalmimon

I have tried Object.getOwnPropertyNames() but that doesn't include the instance proerties

Yes, that's because instances are not defined on the parser structure object itself (aka an instance of the user-defined type transpiled into JavaScript as a function (which can be instantiated, so it behaves like a "class")), but on its object prototype (more precisely they're primarily defined on the function prototype, but the new operator sets that as the prototype of the object being created). Normally you don't care about whether a property is set directly on the object or its prototype, because usual operations like obj.key/obj['key']/for (let prop in obj) { ... } all traverse the prototype chain, so you'll see there even properties from the prototype and also from the prototype of the prototype, if it exists :). See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Inheritance_and_the_prototype_chain.

But Object.get*Own*PropertyNames() is exactly one of the functions that does not go through the prototype chain, so you'll get only properties that are directly set on the object queried.

1 reply

Mundra Pavani

@MundraPavani

Hi, Good evening, can someone help me to this solve error import error cannot import kaitaistructerror

oldbug

@oldbug:matrix.org

[m]

That’s most likely because you are using an „outdated“ ksc

Mundra Pavani

@MundraPavani

i am using kaitai struct compiler version 0.9

oldbug

@oldbug:matrix.org

[m]

Show your error message pls

The work you’ve done was based on a PyPI release installation - that’s really hard to solve…

At least what I’ve seen so far

Mundra Pavani

@MundraPavani

how can i attach a image here ?

oldbug

@oldbug:matrix.org

[m]

Can’t you paste the console output?

mfalkvidd (Mikael Falkvidd): nice to see you in here 😊👍

Mundra Pavani

@MundraPavani

https://ibb.co/w6LLpLG

please find the image in this link

oldbug

@oldbug:matrix.org

[m]

Yeah, that is definitely an error thrown by a ks runtime <ksc 0.9

Mundra Pavani

@MundraPavani

how can we solve it ?

oldbug

@oldbug:matrix.org

[m]

This error handling has been introduced after we updated to ksc 0.9 in satnogs-decoders

dgelessus

@dgelessus

Do you have a file called "kaitaistruct.py" in the same folder as your other code? If so, you need to rename that, or it will conflict with the kaitaistruct module.

(to clarify: kaitaistruct should be installed through pip, so there shouldn't be a kaitaistruct.py in your own code)

oldbug

@oldbug:matrix.org

[m]

the whole story short: librespacefoundation/satnogs/satnogs-decoders!207

so everything after version 1.26.2 of the satnogs-decoders package needs to be compiled with a KSC >= 0.9

oldbug

@oldbug:matrix.org

[m]

it is probably as easy as running ./contrib/docker-ksc.sh

inside the root directory of the source

...of satnogs-decoders

Mundra Pavani

@MundraPavani

i will try this

thank you

oldbug

@oldbug:matrix.org

[m]

I'm not able to be around for much longer, so: did you make any progress, @MundraPavani ?

Mundra Pavani

@MundraPavani

I am just facing this Kaitai struct error issue

oldbug

@oldbug:matrix.org

[m]

ok, there is another issue

an additional one

try installing pip install kaitaistruct~=0.9.0

as seen in the logs, with ~0.8.0 exactly your error is shown! so you are most probably using an old installation of satnogs-decoders, too

the logs:

❯ pip install -e .
Obtaining file:///Users/pd/Documents/software/satnogs-decoders
  Preparing metadata (setup.py) ... done
Collecting kaitaistruct~=0.8.0
  Using cached kaitaistruct-0.8-py2.py3-none-any.whl
Requirement already satisfied: enum34~=1.1.0 in /Users/pd/Documents/software/.decoders-venv/lib/python3.8/site-packages (from satnogs-decoders==1.28.1+1.g4db03c4.dirty) (1.1.10)
Installing collected packages: kaitaistruct, satnogs-decoders
  Attempting uninstall: kaitaistruct
    Found existing installation: kaitaistruct 0.9
    Uninstalling kaitaistruct-0.9:
      Successfully uninstalled kaitaistruct-0.9
  Attempting uninstall: satnogs-decoders
    Found existing installation: satnogs-decoders 1.28.1+1.g4db03c4
    Uninstalling satnogs-decoders-1.28.1+1.g4db03c4:
      Successfully uninstalled satnogs-decoders-1.28.1+1.g4db03c4
  Running setup.py develop for satnogs-decoders
Successfully installed kaitaistruct-0.8 satnogs-decoders-1.28.1+1.g4db03c4.dirty
❯ decode_multiple Ax25monitor --filename ../sampledata/iss/25544-293-20200917T095620Z-month.csv --format csv
Traceback (most recent call last):
  File "/Users/pd/Documents/software/.decoders-venv/bin/decode_multiple", line 33, in <module>
    sys.exit(load_entry_point('satnogs-decoders', 'console_scripts', 'decode_multiple')())
  File "/Users/pd/Documents/software/.decoders-venv/bin/decode_multiple", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/pd/Documents/software/satnogs-decoders/satnogsdecoders/decode_multiple.py", line 15, in <module>
    from kaitaistruct import KaitaiStructError
ImportError: cannot import name 'KaitaiStructError' from 'kaitaistruct' (/Users/pd/Documents/software/.decoders-venv/lib/python3.8/site-packages/kaitaistruct.py)

so it is not the compiler but the python runtime version instead that throws that error

oldbug

@oldbug:matrix.org

[m]

❯ git checkout 1.26.2
Note: switching to '1.26.2'.
[snip]
HEAD is now at ee882c9 Catch KaitaiStructErrors instead of raising them
❯ cat setup.cfg
[metadata]
name = satnogs-decoders
url = https://gitlab.com/librespacefoundation/satnogs/satnogs-decoders
[snip]
[options]
packages = find:
include_package_data = True
install_requires =
    kaitaistruct~=0.9.0
[...]

so your source or PyPI installation is older than 1.26.2

Where communities thrive

People

Repo info

Activity