Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Hirad Yazdanpanah
    @hiradyazdan
    @AArnott there’s an issue for it already neuecc/Utf8Json#46 i have commented at th end.
    Hirad Yazdanpanah
    @hiradyazdan
    the suggestion in the issue, seems a bit overkill as one might not need to define separate assemblies on unity for example, for only different serializers where both share the same core logic
    Hirad Yazdanpanah
    @hiradyazdan
    @AArnott is there anyway to ignore null values from a serialized object in int key mode? I understand that int keys will set null if value not provided or indexes are skipped, as the receiver would be accepting the same key index, but what if sender has different indexing than receiver? say in two different environments where messagepack-csharp is used for the serializer only?
    Andrew Arnott
    @AArnott
    I'm not sure I understand your question, @hiradyazdan. In int key mode, MessagePack stores each field as an element in an array. If we didn't store a messagepack nil code in a slot of that array, what would we store instead? If we "compressed" it by simply omitting those elements then all the indexes would be off and the deserializer wouldn't know how to interpret each element.
    Hirad Yazdanpanah
    @hiradyazdan
    @AArnott can’t we skip the nil and reindex? let’s say in the IL or the generated code we reorder the elements if they have nil values before serializing to binaries
    the use case of this would be when there are two different model structures required for the same model, where one doesn’t need the elements that the other need (aiming to two different servers for example)
    Hirad Yazdanpanah
    @hiradyazdan
    if we could have an option that by choice we reindex the elements and exclude the nil values this scenario could be possible
    Hirad Yazdanpanah
    @hiradyazdan
    otherwise, you’d have to have an extra copy of the same model and its properties with different keys only, which is excessive from developers’ point of view!
    Andrew Arnott
    @AArnott
    @hiradyazdan: let's say you have a class with 4 properties that you're deserializing, and you receive a msgpack binary with ['a','d','g']. One of the properties are missing because it was nil. But which one was it?
    Hirad Yazdanpanah
    @hiradyazdan
    @AArnott oh i forgot to clarify, let’s say the deserializer is on another environment with different model structure which expects 3 properties already
    you have a deserializer using msgpack-csharp in c# env which can work fine without skipping nils and reindxing, and another one, on another server for example, that expects different model structure, which skips and reindex.
    Hirad Yazdanpanah
    @hiradyazdan
    i mean the class/struct which defines the properties on the other server only correspond to some of the properties in the first one, not all
    Andrew Arnott
    @AArnott
    It sounds like you have two potential schemas you want to serialize to. You can do that in either of two ways: The simplest is to define each schemas in terms of a C# class. Serialize an instance of the class based on where you're sending the serialized result. So if you have one destination that expects 3 properties and one that expects 4, instantiate and initialize both classes, and serialize each one.
    The other option is to write a custom formatter that knows to skip specific properties and serialize with that one when you want to omit them.
    Hirad Yazdanpanah
    @hiradyazdan
    @AArnott i like the second option, is there any example for this scenario already?
    Hirad Yazdanpanah
    @hiradyazdan
    when you say know to skip specific properties, do you mean it can only know by property name or its value? cuz if it’s by the name then wouldn’t cause a conflict for the other formatter where it needs those properties?
    Andrew Arnott
    @AArnott
    Of course it wouldn't skip based on value or else again the deserializer won't know which array index to deserialize into which property.
    pmArchitect
    @pmArchitect
    Is there a simple tutorial or step-by-step instructions on how to use MessagePack in Unity?
    Or an Example Project?
    Kevin Streicher
    @NoxMortem
    @pmArchitect There is nothing specific about MessagePack and Unity. Use it as the first code block in the readme on the repository shows. The ONLY thing unity specific is, that it can serialize some of the types of Unity. You still need to separete serializeable and non-serialiezable data and assign the values deserialized after loading to whatever you use in Unity.
    Andrew Arnott
    @AArnott
    @neuecc I think I'll release a stable 1.8 version so we can service it more easily than we can service the v1.7 branch. Any concerns?
    Andrew Arnott
    @AArnott
    I've released a stable 1.8 to nuget.org.
    @neuecc With #589 I think we've made all the public API and behavioral changes we want to for 2.0. How do you feel about a stable package?
    I'm going to go through the active PRs and maybe a few of the issues to see if any are best done before 2.0 stabilizes.
    (and probably aggressively close some stale PRs)
    Yoshifumi Kawai
    @neuecc
    @AArnott Yes, just I've completed working for IL2CPP. #591, #592
    Jakub Holovsky
    @JakubHolovsky
    Question - does Messagepack "mpc" support the new PackageReference attribute in csproj? A year ago it was not true so I wonder if the support has been added? Cheers
    Jon Steinich
    @jsteinich
    @JakubHolovsky it depends what version of mpc you are using. The universal code generator version does work, but it is slated to be deprecated. Some of the newest non-cross-platform versions also worked.
    Andrew Arnott
    @AArnott
    @neuecc I'm looking at your PR now. It's great to see several issues fixed by it.
    I notice that you were upset by the 'lint' causing PR build breaks. I'm happy to fix them, but I just wonder why they only hit you in the CI build. Don't you see them as build warnings locally?
    Yoshifumi Kawai
    @neuecc
    @AArnott thanks for looking. About lint, maybe #556, during working for Unity, I've deleted Directory.Build.props.
    Turik
    @turik441
    Hello.
    Can MessagePack c # deserialize an array of objects lazy?
    @neuecc Hello. Can MessagePack deserialize an array of objects lazy?
    Andrew Arnott
    @AArnott
    If you mean can it deserialize a Lazy<int>[], I think not. Even if it could, the values inside wouldn't actually be lazy -- they would have to be constructed at deserialization time. Unless perhaps you were willing to keep the ReadOnlySequence<byte> that you deserialized from around indefinitely.
    Turik
    @turik441
    @AArnott
    @AArnott I have a very large array of objects, when deserializing I can not load them into memory. I would like to use something like .deserialize <IEnumerable <myClass>>. And then the myClass object was deserialized only by accessing it in the foreach body
    Andrew Arnott
    @AArnott
    With some effort that could probably be done, again assuming you don't mind keeping the serialized form in memory while the enumerable isn't fully realized.
    But in order for the contents of your enumerable to have been serialized in the first place, it would have had to all be in memory. So is this even a real thing? If they can't all fit in memory at once, how did you serialize it in the first place?
    Turik
    @turik441
    @AArnott I serialized an array of objects using lazy evaluation
    Andrew Arnott
    @AArnott
    You had an array? So what, was it Lazy<T>[] ? And now you want to deserialize it lazily using IEnumerable<T>? Regardless, I imagine there's a few ways to do it, all of which requires that you hold the ReadOnlySequence<byte> that you deserialize from (or at least parts of it) in memory indefinitely (or until all elements have been fully realized).
    But IMO you should probably think about this differently. If you have a very large object graph, consider serializing it separately, and index it, so that you can look up that object and deserialize it on-demand when calling a particular method to access it. That way one person enumerating your lazily constructed collection won't permanently allocate a ton of memory in your process.
    Andrew Arnott
    @AArnott
    @turik441 I recently came across #109 which appears to be similar to your feature request. I added a comment there that explains how we might accomplish it (and how I'm accomplishing it already via some custom formatters I use in my app).
    @neuecc I propose we're ready to release 2.0 as a stable package. Thoughts or concerns?
    Yoshifumi Kawai
    @neuecc
    @AArnott No, when dependency on Nerdbank.Streams. so I've made PR #636 .
    Andrew Arnott
    @AArnott
    @neuecc Now that your PR has completed, any concerns with 2.0 going stable?
    Andrew Arnott
    @AArnott
    @neuecc ping? Also do you want more time to respond to my comments on #642?
    Yoshifumi Kawai
    @neuecc
    @AArnott Ah, sorry, I've missed. Ok to going 2.0 stable.
    However, if stable means not rc, we should require to publish mpc, unitypackage, too.
    Yoshifumi Kawai
    @neuecc

    By the way, v2 isn't compatible with streaming reading and writing, right?

    For writing, we use a chunk link that borrows a buffer from BufferPool,
    It is not possible to write by Streaming.
    (For IBufferWriter, Flush is not called, so a buffer is created internally)

    As for reading, since it is necessary to convert to ReadOnlySequence once, it is not read by streaming.

    In other words, A buffer of that size(if data is too large, require large buffer) is required for both writing and reading.

    This is not a concern.
    I just thought about an API that combines high-speed reading and writing with Streaming.
    That hasn't been achieved yet.

    Andrew Arnott
    @AArnott
    Thanks, @neuecc . And yes, by going stable, I mean shedding the -rc or any other suffix so that NuGet/SemVer considers the package stable, and (just as important) we consider the public API stable so that future changes do not break backward compatibility unless we're willing to ship a 3.0 release.
    Andrew Arnott
    @AArnott
    Regarding streaming, MessagePack v2.0's streaming support is to asynchronously drain the stream to memory buffers then deserialize all at once, or to serialize everything to memory buffers then asynchronously copy it all to streams. Of note though is this does not require one contiguous buffer. All the serializing or reading from a stream is to a Sequence<T> which allocates many moderately sized arrays so that they don't go to the large object heap, and you don't run out of memory just because you can't allocate a large enough array. If #437 is ever done (set for 3.0 currently) it will allow asynchronous streaming deserialization without sacrificing performance.
    IMO though, buffering msgpack before (de)serialization to an async stream isn't nearly as bad when it is broken up into many smaller arrays as 2.0 does. An original or deserialized object graph is very likely to take up much more memory than the msgpack buffers that represent it, and those buffers often get recycled too. So until someone complains about not being able to serialize some very large object graph because of lack of streaming serialization, I wouldn't worry about it.

    However, if stable means not rc, we should require to publish mpc, unitypackage, too.

    How close are we to publishing mpc / unitypackage? Is there an issue tracking this? Is #593 tracking something like this?

    Yoshifumi Kawai
    @neuecc
    Thank you for your explanation.
    Yes, it is the same as my thought, I've relieved.