Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Ismael Hamed
    @ismaelhamed
    The index gap sample included in the README.md works, but if I try to make the class "immutable", I can't seem to figure out how to set up the constructor to make it work. No matter what I do I always get MessagePackDynamicObjectResolverException: can't find matched constructor.
    [MessagePackObject]
    public class IntKeySample
    {
        [Key(3)]
        public int A { get; }
        [Key(10)]
        public int B { get; }
    
        [SerializationConstructor]
        public IntKeySample(int a, int b)
        {
            A = a;
            B = b;
        }
    }
    Max Wagner
    @emeyex
    @AArnott Can you try suppressing deployment of your own System.Memory.dll and see if that fixes the issue I'll have to investigate how this is done... still getting up to speed on .NET subtleties. If you've got more details at the ready I'll take 'em, otherwise I'll look into it. Thanks.
    Andrew Arnott
    @AArnott
    @ismaelhamed Unfortunately deserializing constructors with ordinal keys have to have a parameter for every key position, and each parameter must match a member. You can change the members to object and leave them null though. Otherwise the only other option is write your own IMessagePackFormatter<T> for your type.
    @emeyex I have no pointers, other than just try deleting the file from the app directory and see if it works.
    Ismael Hamed
    @ismaelhamed
    @AArnott oh I see, thanks!
    Omer Raviv
    @OmerRaviv
    Hey all. Thank you for the wonderful library and for having such a nice community here. I have a complete n00b question. I've been reading up and trying to wrap my head around IBufferWriter, System.Pipelines, etc, but I can't seem to find an answer to a very simple question - what syntax should I use if I want to write a MessagePack payload via HTTP using HttpClient / HttpWebRequest, without incurring the cost of copying the buffer over to the underlying stream?
    Andrew Arnott
    @AArnott
    @OmerRaviv These web APIs give you a request Stream, correct? You can then use the UsePipeWriter extension method to get a PipeWriter for use in writing to that Stream. Since PipeWriter implements IBufferWriter<byte>, you can pass it directly to MessagePackSerializer.Serialize.
    Although in effect this is what our built-in overload does, so is there a reason not to use that?
    Note that neither of these actually streams serialization out. It makes effective use of many small arrays as buffers and then asynchronously streams them out. But to get actual serialize-more-than-you-have-memory-for-because-you're-transmitting-as-you-go behavior requires more work. Folks have done it with this library, but it's a little ugly, and almost never necessary.
    Kevdome3000
    @Kevdome3000
    Hey all, I'm using MessagePack for mobile game development in Unity (mostly iOS so AOT) and I've been able to get away with mostly just using MessagePackObject attribute and haven't had to create any custom formatters/ resolvers. That said, I'd like to use the Union feature but can't figure it out for the life of me. As an Example:
        [Union(2, typeof(Mission))]
        [Union(1, typeof(Challenge))]
        [Union(0, typeof(Quest))]
        [Serializable, MessagePackObject(true), Table("Tasks")]
        public abstract class Task : IGameReadModel
        {
            [PrimaryKey, CanBeNull] public string InstanceId { get; set; }
    
            public TaskType? TaskType { get; set; }
    
            public TaskLength? TaskLength { get; set; }
    
            [CanBeNull] public string TargetId { get; set; }
    
            [CanBeNull] public string StatId { get; set; }
    
            [Ignore, CanBeNull] public string Type { get; set; }
    
            public string Title { get; set; }
    
            public string Description { get; set; }
    
            public double Current { get; set; }
    
            public double? Avg { get; set; }
    
            public double Target { get; set; }
    
            public string Condition { get; set; }
    
        }
    
        public class EmptyTask : Task
        {
        }
    
        [Serializable, MessagePackObject(true)]
        public class Quest : Task
        {
            public string MilestoneId { get; set; }
        }
    
        [Serializable, MessagePackObject(true)]
        public class Challenge : Task
        {
            public ChallengeType? ChallengeType { get; set; }
    
        }
    
        [Serializable, MessagePackObject(true)]
        public class Mission : Task
        {
            public MissionType? MissionType { get; set; }
        }
    As a test, I call it like this:
          var challenge = item.CustomData[TaskType.Challenge.ToString()];
    
            var serialized = MessagePack.MessagePackSerializer.Serialize(challenge);
    
            var bin = serialized.Utf8();
            Debug.Log(bin);
            var deserialized = MessagePackSerializer.Deserialize<Task>(serialized);
    and I get this error:
    MessagePack.MessagePackSerializationException : Failed to deserialize Chance.Domain.Game.Domain.Infrastructure.Task value.
      ----> MessagePack.MessagePackSerializationException : Unexpected msgpack code 218 (str 16) encountered.
    Kevdome3000
    @Kevdome3000

    but the Debug shows this:

    ��{"InstanceId":null,"TaskType":null,"TaskLength":null,"ChallengeType":null,"TargetId":"#####","StatId":"####","Type":"######","Title":"#####","Description":"#######","Current":235,"Avg":0.34,"Target":45,"Condition":">="}

    so I can see the data is there, but can't figure out why it's throwing an error

    What am I missing? Also, any advice on how best to debug these kind of scenarios would be much appreciated. Thanks in advanced!
    Andrew Arnott
    @AArnott

    @Kevdome3000 Your problem is that you're serializing a Challenge as a Challenge but then deserializing the base class of Task. You must serialize and deserialize with the same static type. So for example if you call Serialize<Challenge> (which you do in your code snippet, although the generic type argument is inferred by C#) then you must call Deserialize<Challenge>. Challenge is not a Union type and thus it is serialized without any union header.
    But if you want/need to deserialize a Task, then you should serialize a Task, like so:

    MessagePackSerializer.Serialize<Task>(challenge);

    Note that the object you're passing in is still an instance of type Challenge but the generic type argument you specify (explicitly!) is the base type -- the union type. This way, the Task formatter will be used and it will include the union header, and then go on to serialize the Challenge object itself (including the unique Challenge.ChallengeType field). Then when you deserialize a Task, the union header it requires will be there, it will recognize that the actual object type is Challenge and everything will be happy.

    Kevdome3000
    @Kevdome3000
    @AArnott Ah yes of course, that makes sense. Thank you!
    Mark
    @marknasr22

    Hello, I am getting the following error (I am using Unity):
    MessagePackSerializationException: Unexpected msgpack code 230 (negative fixint) encountered.
    The error is caused when deserializing the following class:

    [MessagePackObject]
    public class SimpleMessage
    {
        [Key(0)]
        public string message;
        [Key(1)]
        public int imageIndex = -1;
        [Key(2)]
        public bool hasGifID = false;
    
        [Key(3)]
        public MessageDate messageDate;
    
        [Key(4)]
        public uint sender;
    }

    Here is MessageDate:

    [MessagePackObject]
    public class MessageDate
    {
        [Key(0)]
        public int year;
        [Key(1)]
        public int month;
        [Key(2)]
        public int day;
        [Key(3)]
        public int hour;
        [Key(4)]
        public int minute;
        [Key(5)]
        public int second;
    }
    Mark
    @marknasr22
    It doesn't always happen and I can reproduce this error in the Editor.
    Andrew Arnott
    @AArnott
    What data are you deserializing, @marknasr22 ? Your types look fine, but the exception you mention is thrown when the data being deserialized doesn't match. It would also help to see the callstack of the exception.
    Mark
    @marknasr22

    @AArnott Thank you for responding.

    Now I am getting code 226

    MessagePackSerializationException: Unexpected msgpack code 226 (negative fixint) encountered.
    MessagePack.MessagePackReader.ThrowInvalidCode (System.Byte code) (at Assets/Scripts/MessagePack/MessagePackReader.cs:942)
    MessagePack.MessagePackReader.TryReadMapHeader (System.Int32& count) (at Assets/Scripts/MessagePack/MessagePackReader.cs:444)
    MessagePack.MessagePackReader.ReadMapHeader () (at Assets/Scripts/MessagePack/MessagePackReader.cs:387)
    MessagePack.Formatters.DictionaryFormatterBase`5[TKey,TValue,TIntermediate,TEnumerator,TDictionary].Deserialize (MessagePack.MessagePackReader& reader, MessagePack.MessagePackSerializerOptions options) (at Assets/Scripts/MessagePack/Formatters/DictionaryFormatter.cs:84)
    MessagePack.Formatters.OperationDataFormatter1.Deserialize (MessagePack.MessagePackReader& reader, MessagePack.MessagePackSerializerOptions options) (at <2dda8b71668f401ab737424e74148583>:0)
    MessagePack.MessagePackSerializer.Deserialize[T] (MessagePack.MessagePackReader& reader, MessagePack.MessagePackSerializerOptions options) (at Assets/Scripts/MessagePack/MessagePackSerializer.cs:246)
    Rethrow as MessagePackSerializationException: Failed to deserialize OperationData value.
    MessagePack.MessagePackSerializer.Deserialize[T] (MessagePack.MessagePackReader& reader, MessagePack.MessagePackSerializerOptions options) (at Assets/Scripts/MessagePack/MessagePackSerializer.cs:251)
    MessagePack.MessagePackSerializer.Deserialize[T] (System.ReadOnlyMemory`1[T] buffer, MessagePack.MessagePackSerializerOptions options, System.Threading.CancellationToken cancellationToken) (at Assets/Scripts/MessagePack/MessagePackSerializer.cs:270)
    Serializer.Deserialize[T] (System.ArraySegment`1[T] buffer) (at Assets/Scripts/Server-Client/ClientSetup.cs:154)

    This is what I have on ClientSetup.cs:154:

    var resolver = MessagePack.Resolvers.CompositeResolver.Create(
                MessagePack.Resolvers.StandardResolver.Instance,
                MessagePack.Resolvers.ContractlessStandardResolver.Instance,
                MessagePack.Resolvers.BuiltinResolver.Instance,
                MessagePack.Resolvers.PrimitiveObjectResolver.Instance
               );
    
            Options = MessagePackSerializerOptions.Standard.WithResolver(resolver);
    
    ArraySegment<byte> buffer //This is an ArraySegment that I get from the server after it is serialized
    
    MessagePackSerializer.Deserialize<SimpleMessage>(buffer, Options);

    Here is serialization of the server:

    MessagePackSerializer.Serialize<SimpleMessage>(simpleMessage, Options)
    Andrew Arnott
    @AArnott
    It looks to me like your code is fine. So I suspect data corruption. During deserialization, an integer is found where a map is expected.
    Cognitive X Solutions
    @cognitivex
    I have the following structure , class MyClass , List<ICommand> commands, I get an exeception when using message pack, MessagePack.Resolvers.TYpelessContractlessResolver --> messagepack.Formatternotregisteredexception for ICommand
    anyway arround this?
    Andrew Arnott
    @AArnott
    @cognitivex ICommand doesn't tend to be something you want to or can serialize. If you want to skip it, you can add [IgnoreDataMember] or the messagepack-specific equivalent to that property.
    Artem
    @codename-art

    Hi all! If I understand correctly, the collection formatters only serialize the elements of the collection, not the fields / properties of the collection.
    This is a simplified example:

    [MessagePackFormatter(typeof(GenericCollectionFormatter<Email, ComplexEmailsList>))]
        public class ComplexEmailsList : List<Email>
        {
            public ComplexEmailsList()
            {
            }
            public string Organization;
        }

    Will the Organization value be serialized and deserialized? And is it possible to do it?

    Andrew Arnott
    @AArnott
    @codename-art Correct: the collection formatters all write out a msgpack array header with the number of elements in the collection, followed by the elements of the collection.
    For your special collection type, you'll probably want to write your own formatter. I imagine you'll have a doubly-nested array: one array for your overall ComplexEmailsList type with two elements: [Organization, []] where the second element is the nested array for the actual collection elements.
    Simonas Baltulionis
    @simonasB

    Hi,

    First of all, thank you for your great work developing this library, and happy new year ( a bit late but the first post in the new year).
    I want to ask for opinions and recommendations about my use case. I have a custom type that consists of only primitives (int, double, string). I need to serialize a collection of such items to the file (~3 million, the serialized binary size is ~200 MB). The serialization happens not often, so the performance can be ignored. The most important part is the deserialization and I want to optimize it as much as possible (memory-wise and time-wise).

        [MessagePackObject]
        public class MyType
        {
            [Key(0)] public uint Field1;
            [Key(1)] public double Field2;
            [Key(2)] public string Field3;
            //....
        }
    
        private static void CreateMessagePackFile()
        {
            var items = new List<MyType>(); // filled collection
            using (var fs = new FileStream("items.bin", FileMode.Create, FileAccess.Write))
            {
                MessagePackSerializer.Serialize(fs, items);
            }
    
        }
    
        private static void LoadMessagePackFile()
        {
            using var stream = File.OpenRead("items.bin");
    
            var items = MessagePackSerializer.Deserialize<List<MyType>>(stream);
        }
    1. Is this the most efficient way to do it? I need all items to be in memory because the collection will be constantly used and iterated.

    2. Another question is about the lazy deserialization of fields. Some string fields are very rarely used. Is it possible somehow to deserialize the field only when it's accessed? Something like the ZeroFormatter had. I saw your comments about writing a custom formatter for this use case. Could you tell me is this really possible and maybe share some code snippet on how to do it?

    Thanks.

    3 replies
    ModernRonin
    @ModernRonin
    how can I use the source generator (mpc) with the TypelessContractlessStandardResolver? the documentation seems to indicate that mpc works only with annotated types? in my scenario this is not really feasible, I got literally hundreds of types most of which has 1-2 dozen properties.
    Kevin Streicher
    @NoxMortem
    @AArnott , @neuecc do you have any plans to support the new types from Unity.Mathematics (float4x4, ...) and Unity.Physics (Aabb,...)?
    Kevin Streicher
    @NoxMortem

    And if not, how do I use MessagePack.Generator (mpc.exe) with CustomResolver?

    dotnet mpc -ei "System.Ulid" -i . -o "Assets\Scripts\Frameworks\MessagePack"
    Project Compilation Start:CodeGenTemp
    Project Compilation Complete:00:00:26.9081418
    Method Collect Start
    Fail in console app running on MessagepackCompiler.RunAsync
    MessagePackCompiler.CodeAnalysis.MessagePackGeneratorResolveFailedException: Serialization Object must mark MessagePackObjectAttribute. type: global::Unity.Mathematics.float4x4

    I have written a CustomFormatter

    public sealed class Float4x4Formatter : global::MessagePack.Formatters.IMessagePackFormatter<global::Unity.Mathematics.float4x4>

    And tried to use it via [MessagePackFormatter(typeof(ArrayFormatter<float4x4>))]

    [Key(6)]
    [MessagePackFormatter(typeof(ArrayFormatter<float4x4>))]
    public float4x4[] LocalToWorld;

    But I am not sure how to make sure the CustomFormatter is discovered.

    Kevin Streicher
    @NoxMortem
    Ah. What a misconception. Obviously you simply do not, but exclude it via dotnet mpc -ei "System.Ulid,Unity.Physics.Aabb,Unity.Mathematics.float4x4" and register it manually. UlidMessagePackFormatter is a good example on how to do it.
    Andrew Arnott
    @AArnott
    I'm glad you figured it out. I don't use Unity personally, so I can't make much in the way of recommendations there.
    Kevin Streicher
    @NoxMortem
    @AArnott yeah. I somehow got stuck in my own thinking. Looking at the UlidMessagePackFormatter made it much more clear what the readme.md anyway tried to tell me.
    Alok Sharma
    @aloksharma1
    Hello All, i am trying to use messagepack for serializing my disk cache values
    but on desrialize it gives me error
    fail: Microsoft.AspNetCore.Server.Kestrel[13]
          Connection id "0HMH52VP9QPUC", Request id "0HMH52VP9QPUC:00000023": An unhandled exception was thrown by the application.
          MessagePack.MessagePackSerializationException: Failed to deserialize Ultra.CommonFramework.Caching.Faster.FastCacheEntries`1[[System.Collections.Generic.List`1[[Ultra.CachingModule.Domain.CachedPageData, Ultra.CachingModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] value.
           ---> System.NotSupportedException: Specified method is not supported.
             at Microsoft.Extensions.Primitives.StringValues.System.Collections.Generic.ICollection<System.String>.Add(String item)
             at MessagePack.Formatters.GenericCollectionFormatter`2.Add(TCollection collection, Int32 index, TElement value, MessagePackSerializerOptions options)
             at MessagePack.Formatters.CollectionFormatterBase`4.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.Formatters.KeyValuePairFormatter`2.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.Formatters.ListFormatter`1.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.Formatters.Ultra_CachingModule_Domain_CachedPageDataFormatter2.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.Formatters.ListFormatter`1.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.Formatters.Ultra_CommonFramework_Caching_Faster_FastCacheEntries`1\[\[System_Collections_Generic_List`1\[\[Ultra_CachingModule_Domain_CachedPageData\, Ultra_CachingModule\]\]\, System_Private_CoreLib\]\]Formatter1.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
             --- End of inner exception stack trace ---
             at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
             at MessagePack.MessagePackSerializer.Deserialize[T](ReadOnlyMemory`1 buffer, MessagePackSerializerOptions options, CancellationToken cancellationToken)
    this is my data holder class
    [MessagePackObject]
        public class CachedPageData
        {
            [Key(0)]
            public CompressionType CompressionType { get; set; }
            [Key(1)]
            public byte[] ResponseData { get; set; }
            [Key(2)]
            public bool IsMinified { get; set; }
            [Key(3)]
            public List<KeyValuePair<string, StringValues>> Headers { get; set; }
        }
    is using byte array not supported?
    Alok Sharma
    @aloksharma1
    solved it seems keyvaluepair & StrinValues class were the issue
    Andrew Arnott
    @AArnott
    👍
    glitch
    @glitchland
    Hi folks, I'm running into a strange issue where the following data is failing deserialization:
    \x84\xA3\x41\x53\x47\x00\xA3\x41\x53\x49\x93\x90\x90\x90\xA3\x41\x53\x53\x93\x00\x00\x00\xA3\x41\x52\x53\x93\x00\x00\x00
    The error is: MessagePack.MessagePackSerializationException: Failed to deserialize WSVJStateMsg value. ---> MessagePack.MessagePackSerializationException: Unexpected msgpack code 144 (fixarray) encountered.
    [MessagePackObject(keyAsPropertyName: true)]
    public class WSVJStateMsg {
    public int ASG { get; set; } // ActiveSceneGroup
    public int[,] ASI { get; set; } // ActiveSceneItems
    public float[] ASS { get; set; } // ActiveScrollerSpeed
    public float[] ARS { get; set; } // ActiveRotatorSpeed
    }
    Apologies for the poor naming, I'm just wondering why this error would trigger?
    This is how I deserialize: WSVJStateMsg wvjsm = MessagePackSerializer.Deserialize<WSVJStateMsg>(vjStateMsg.D);
    Alok Sharma
    @aloksharma1
    @glitchland what is WSVJStateMsg ?
    i mean the sample data, when you do serialization
    did you tried different messagepack policy for serialization and de-serialization
    Punit Sharma
    @punitsharma:matrix.org
    [m]
    This message was deleted
    2 replies
    www-prog
    @www-prog

    I have a question about the SerializationConstructor.
    Even if there is no constructor that accepts all parameters, it works without [SerializationConstructor].
    This is an example that works.

    [MessagePackObject(true)]
    public record Test
    {
        public string A { get; init; } = string.Empty;
        public string B { get; init; } = string.Empty;
        public string C { get; init; } = string.Empty;
    
        //[SerializationConstructor] no need
        public Test(string a)
        {
            A = a;
            B = "No B";
            C = "No C";
        }
    }

    If param is of a different type, construct must be specified. In this case, using [SerializationConstructor] does not solve the problem.
    This is an example that doesn't work.

    [MessagePackObject(true)]
    public record Test
    {
        public string A { get; init; } = string.Empty;
        public string B { get; init; } = string.Empty;
        public string C { get; init; } = string.Empty;
    
        [SerializationConstructor]
        public Test(int a)
        {
            if (a == 1) A = "A is 1";
        }
    }

    Why are you adding [SerializationConstructor]?