by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    kaeraz
    @kaeraz
    I cannot create LEN2 field in above way. Can you please help me?
    Arkadiusz Bulski
    @arekbulski
    Const wont work because it requires a constant value.
    Maybe write your own class for this?
    kaeraz
    @kaeraz
    ok, thanks
    Is there any build-in solution?
    I have another question. I want to use Checksum(...) for calculatin crc. How can I tell the Checksum to calculate it from the stream starting from e.g. LEN1 and finishing at LEN2, so that crc is calculated from 4 bytes?
    Arkadiusz Bulski
    @arekbulski
    There is no existing class for this I think.
    Checksum takes data from another field, not from the stream. Use a Peek(Bytes) or something, or write a different Checksum implementation.
    I also suggest that you post your questions as Issues, there is more people watching those than watching the chat I think.
    Arkadiusz Bulski
    @arekbulski
    Ah I forgot... instead of Const check out the Rebuild class.
    kaeraz
    @kaeraz
    ok, thanks, I will take a look.
    kaeraz
    @kaeraz
    thanks for Rebuild class hint, it does the job
    butterwagon69
    @butterwagon69

    Hey there! I'm working on a parser for Unreal Engine packages in python and I think Construct would be a great choice. I'm looking at parsing an "Index" field, which is of variable length and where a bit from each byte is used to indicate if there are more bytes in the index. Here's some naive python code that does what I'm talking about:

    def read_idx(stream):
        byte0 = read_byte(stream)
        result = 0
        if byte0 & 0x40:
            byte1 = read_byte(stream)
            if byte1 & 0x80:
                byte2 = read_byte(stream)
                if byte2 & 0x80:
                    byte3 = read_byte(stream)
                    if byte3 & 0x80:
                        byte4 = read_byte(stream)
                        result = byte4
                    result = (result << 7) | (byte3 & 0x7F)
                result = (result << 7) | (byte2 & 0x7F)
            result = (result << 7) | (byte1 & 0x7F)
        result = (result << 6) | (byte0 & 0x3F)
        if byte0 & 0x80:
            result = -result
        return result

    So the first bit of the first byte is a sign bit, the second bit of the first byte and the first bit of the second, third, and fourth bytes indicate if there are more bytes to read, and the rest of all the bytes are data that get combined into a signed integer.

    I'm not sure if I would be able to implement this as an Adapter, since I'm not sure that any existing Construct objects can represent this kind of data. Should I implement this as an adapter, and if so, how? If not, what should it be implemented as?

    Thanks!

    Arkadiusz Bulski
    @arekbulski
    You are writing a custim class, not an adapter.
    Look at the implementation of VarInt.
    butterwagon69
    @butterwagon69
    Thanks, I will give it a shot and post results. Docs seem to suggest extreme caution when subclassing Construct. Anything in particular I need to watch out for?
    butterwagon69
    @butterwagon69

    Here's my implementation.

    class Idx(Construct):
        lengths = {0: 6, 1: 7, 2: 7, 3: 7, 4: 8}
        negative_bit = 0x80
    
        @staticmethod
        def _get_data_mask(length):
            return (0xFF ^ (0xFF << length)) & 0xFF
    
        @staticmethod
        def _get_more_bit(length):
            return 1 << length
    
        def _parse(self, stream, context, path):
            result = 0
            sign = 1
            i = 0
            depth = 0
            while True:
                length = self.lengths[i]
                bits = byte2int(stream_read(stream, 1))
                mask = self._get_data_mask(length)
                data = bits & mask
                more = self._get_more_bit(length) & bits
                if (i == 0) and (self.negative_bit & bits):
                    sign = -1
                result |= data << depth
                if not more:
                    break
                i += 1
                depth += length
            return sign * result
    
        def _build(self, obj, stream, context, path):
            if not isinstance(obj, integertypes):
                raise IntegerError("Value is not an integer")
            to_write = obj
            for i in range(5):
                byte = 0
                length = self.lengths[i]
                if i == 0:
                    negative = obj < 0
                    byte |= self.negative_bit * negative
                    if negative:
                        to_write *= -1
                mask = self._get_data_mask(length)
                byte |= to_write & mask
                to_write >>= length
                more_bit = (to_write > 0) and self._get_more_bit(length)
                byte |= more_bit
                byte &= 0xFF
                stream_write(stream, int2byte(byte), 1)
                if not more_bit:
                    break
            return obj

    Thanks for the help and the great library!

    Arkadiusz Bulski
    @arekbulski
    I see you you have implemented something similar to VarInt.
    Good job.
    Arkadiusz Bulski
    @arekbulski
    It would be great if your PRed your unreal formats for the gallery.
    butterwagon69
    @butterwagon69

    Done.

    Another question: I'm trying to define structs that map onto a class hierarchy where there's a base object struct and other child structs that extend it. What's the best way to do this? I'm just storing the args for the base class in a list and then extending the list for subclasses like so...

        ut_object_args = [
            "state_frame" / state_frame,
            "property" / If(this._.sup_index, ut_property[5]),
        ]
        ut_object = Struct(*ut_object_args)
        object_class_field = Struct(*ut_object_args, "super_field" / idx, "next" / idx)
    Arkadiusz Bulski
    @arekbulski
    You are doing it right. You can also access a Struct.subcons field, check Struct ctor about it.
    butterwagon69
    @butterwagon69
    I see. Something like this:
    object_class_field = Struct(*ut_object.subcons, "super_field" / idx, "next" / idx)
    Arkadiusz Bulski
    @arekbulski
    Yes.
    muon IT
    @muonIT_twitter
    Hi,
    maybe someone here can help me or point me in the right direction. I try to run this (5 year old) program https://github.com/mungewell/openDR-Remote and, from what I gathered, it uses construct from before 2.8.
    I already figured out that UBInt16 is now Int16ub. Now I get "TypeError: 'FormatField' object is not callable"
    I installed python-2.7 and with pip2.7 construct-2.10.54.
    I know next to nothing about Python (but know about several other programming languages)
    Thx
    Arkadiusz Bulski
    @arekbulski
    Python 2 is no longer supported
    Post details at Issues
    muon IT
    @muonIT_twitter

    If I take a C program that Kernighan and Richie wrote in the '60 I can compile and run it fine today.
    With python I can't even get a program to run from 5 years ago. And this is a one-file-program.

    I somehow don't get comfortable with python and this will probably stay that way. Oh well.

    Arkadiusz Bulski
    @arekbulski
    Not if your C program referenced external modern (regularly updated) library like construct.
    aaron-foster-wallace
    @aaron-foster-wallace
    hi
    does construct support serialization?
    Arkadiusz Bulski
    @arekbulski
    What do you mean?
    (1) you can build blobs as well as parse them
    (2) classes can be pickled, for use in multiprocessing for example
    jhihn
    @jhihn
    File "/tmp/easy_install-j_T31W/construct-2.10.56/construct/lib/binary.py", line 101, in <genexpr> KeyError: '('
    Can anyone help with that error?
    Arkadiusz Bulski
    @arekbulski
    I see you posted an Issue, look there.
    jhihn
    @jhihn
    Yes, I read the docs some more and it said the github issues are preferred. So I went there. Thank you for your help.
    Arkadiusz Bulski
    @arekbulski
    The reason is that Issues get archived and remain searchable in the future. Chat is just pasta without a search button.
    Omer
    @OmerYe
    Hello!
    First I want to say that I very much like and appreciate the project and the effort!
    Struct('x1' / Flag, 'x2' / If(this.x1, Byte )).build({'x1': False})
    It of course raises KeyError: 'x2'. But as x1 is False why is it necessary as x2 is irrelevant anyway?
    Omer
    @OmerYe
    Is there any other way to do the same (more convenient)?
    Jonathan Snyder
    @jpsnyder
    Looks like it is because Struct tries to run build on every element, even if it doesn't make sense.
    Struct('x1' / Flag, 'x2' / Build(If(this.x1, Byte ), None)).build({'x1': False})
    Not sure why If just doesn't do this for you.
    Did I say Build? I meant Default
    Struct('x1' / Flag, 'x2' / Default(If(this.x1, Byte ), None)).build({'x1': False})
    Arkadiusz Bulski
    @arekbulski
    The Default should make the key not needed for bulding. Open an Issue for this please.
    Arkadiusz Bulski
    @arekbulski
    The IF itself does not make the key not required in the passed dict.