Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Turik
    @turik441
    @AArnott I have a very large array of objects, when deserializing I can not load them into memory. I would like to use something like .deserialize <IEnumerable <myClass>>. And then the myClass object was deserialized only by accessing it in the foreach body
    Andrew Arnott
    @AArnott
    With some effort that could probably be done, again assuming you don't mind keeping the serialized form in memory while the enumerable isn't fully realized.
    But in order for the contents of your enumerable to have been serialized in the first place, it would have had to all be in memory. So is this even a real thing? If they can't all fit in memory at once, how did you serialize it in the first place?
    Turik
    @turik441
    @AArnott I serialized an array of objects using lazy evaluation
    Andrew Arnott
    @AArnott
    You had an array? So what, was it Lazy<T>[] ? And now you want to deserialize it lazily using IEnumerable<T>? Regardless, I imagine there's a few ways to do it, all of which requires that you hold the ReadOnlySequence<byte> that you deserialize from (or at least parts of it) in memory indefinitely (or until all elements have been fully realized).
    But IMO you should probably think about this differently. If you have a very large object graph, consider serializing it separately, and index it, so that you can look up that object and deserialize it on-demand when calling a particular method to access it. That way one person enumerating your lazily constructed collection won't permanently allocate a ton of memory in your process.
    Andrew Arnott
    @AArnott
    @turik441 I recently came across #109 which appears to be similar to your feature request. I added a comment there that explains how we might accomplish it (and how I'm accomplishing it already via some custom formatters I use in my app).
    @neuecc I propose we're ready to release 2.0 as a stable package. Thoughts or concerns?
    Yoshifumi Kawai
    @neuecc
    @AArnott No, when dependency on Nerdbank.Streams. so I've made PR #636 .
    Andrew Arnott
    @AArnott
    @neuecc Now that your PR has completed, any concerns with 2.0 going stable?
    Andrew Arnott
    @AArnott
    @neuecc ping? Also do you want more time to respond to my comments on #642?
    Yoshifumi Kawai
    @neuecc
    @AArnott Ah, sorry, I've missed. Ok to going 2.0 stable.
    However, if stable means not rc, we should require to publish mpc, unitypackage, too.
    Yoshifumi Kawai
    @neuecc

    By the way, v2 isn't compatible with streaming reading and writing, right?

    For writing, we use a chunk link that borrows a buffer from BufferPool,
    It is not possible to write by Streaming.
    (For IBufferWriter, Flush is not called, so a buffer is created internally)

    As for reading, since it is necessary to convert to ReadOnlySequence once, it is not read by streaming.

    In other words, A buffer of that size(if data is too large, require large buffer) is required for both writing and reading.

    This is not a concern.
    I just thought about an API that combines high-speed reading and writing with Streaming.
    That hasn't been achieved yet.

    Andrew Arnott
    @AArnott
    Thanks, @neuecc . And yes, by going stable, I mean shedding the -rc or any other suffix so that NuGet/SemVer considers the package stable, and (just as important) we consider the public API stable so that future changes do not break backward compatibility unless we're willing to ship a 3.0 release.
    Andrew Arnott
    @AArnott
    Regarding streaming, MessagePack v2.0's streaming support is to asynchronously drain the stream to memory buffers then deserialize all at once, or to serialize everything to memory buffers then asynchronously copy it all to streams. Of note though is this does not require one contiguous buffer. All the serializing or reading from a stream is to a Sequence<T> which allocates many moderately sized arrays so that they don't go to the large object heap, and you don't run out of memory just because you can't allocate a large enough array. If #437 is ever done (set for 3.0 currently) it will allow asynchronous streaming deserialization without sacrificing performance.
    IMO though, buffering msgpack before (de)serialization to an async stream isn't nearly as bad when it is broken up into many smaller arrays as 2.0 does. An original or deserialized object graph is very likely to take up much more memory than the msgpack buffers that represent it, and those buffers often get recycled too. So until someone complains about not being able to serialize some very large object graph because of lack of streaming serialization, I wouldn't worry about it.

    However, if stable means not rc, we should require to publish mpc, unitypackage, too.

    How close are we to publishing mpc / unitypackage? Is there an issue tracking this? Is #593 tracking something like this?

    Yoshifumi Kawai
    @neuecc
    Thank you for your explanation.
    Yes, it is the same as my thought, I've relieved.

    @AArnott
    Ah, I didn't set up Azure Pipelines (I was going to leave everything to you about CI)

    For mpc, the rest of the work is from this comment
    https://github.com/neuecc/MessagePack-CSharp/issues/617#issuecomment-551076028
    It is also necessary to update the documentation.

    Andrew Arnott
    @AArnott
    Thanks, @neuecc. I've updated the labels on #617 to signify that it's a 2.0 ship blocker and assigned to myself.
    What documentation needs to be updated? I've kept the README file up to date as I've made the various API breaking changes for 2.0, I think.
    Aleksander Heintz
    @Alxandr
    Any reason for the update to System.Collections.Immutable >= 1.6.0 in the latest rc release? It's breaking builds of azure functions.
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : File name: 'System.Collections.Immutable, Version=1.2.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' ---> System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.Assembly.LoadFrom(String assemblyFile) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.AppDomain.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.RuntimeAssembly.GetExportedTypes(RuntimeAssembly assembly, ObjectHandleOnStack retTypes) [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.RuntimeAssembly.GetExportedTypes() [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at MakeFunctionJson.FunctionJsonConverter.TryGenerateFunctionJsons() [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at MakeFunctionJson.FunctionJsonConverter.TryRun() [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :  [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :  [C:\Git\PolicyStaging\src\PolicyStaging\PolicyStaging.csproj]
    C:\Users\D45\.nuget\packages\microsoft.net.sdk.functions\1.0.29\build\netstandard1.0\Microsoft.NET.Sdk
    Yoshifumi Kawai
    @neuecc
    @AArnott about mpc < documantation, and composite resolver for unity.
    Yoshifumi Kawai
    @neuecc
    and how to move v1 to v2(what's different)
    Andrew Arnott
    @AArnott
    @Alxandr No particular reason, except to try to stay a bit more current so folks have the latest libraries via transitive dependencies. Before the upgrade to 1.6.0 I see we were on 1.3.1 which is nearly 3 years old. What is the latest version that wouldn't break Azure Functions?
    Aside: I'm disappointed that Azure Functions breaks given newer assembly versions. Do you know why it's sensitive to this?
    @neuecc Can you write the MPC and unity documentation? I'm much less familiar with those. I can write the v1->v2 migration instructions.
    Aleksander Heintz
    @Alxandr
    @AArnott No clue. I'm also not able to reproduce it in a simple reproduction (Azure/azure-functions-vs-build-sdk#353). That being said, they do have some version issues at runtime due to them bringing their own runtime, but this is happening compile-time which is really confusing. It's failing to load System.Collections.Immutable while doing code-gen.
    Andrew Arnott
    @AArnott
    @Alxandr If you can provide a repro project in a github issue filed against messagepack in the next day or two, I think we can revert the S.C.I. dependency update and keep your issue active to track when we can upgrade it again.
    Aleksander Heintz
    @Alxandr
    @AArnott unfortunately I don't think I can. I'm afraid I'm going to have to debug into msbuild in a really complicated solution with ~70 projects :-/
    And it's work, so I can't just share the code either.
    Aleksander Heintz
    @Alxandr
    Ah. I reproduced it!
    Aleksander Heintz
    @Alxandr
    And issue as you wanted: neuecc/MessagePack-CSharp#654
    Yoshifumi Kawai
    @neuecc
    I'm now reviewing #646, and my rest work is write document of mpc and Unity. And I want to add mpc generate helper for UnityEditor, in v2 release. I'll post All PR within a day.
    Andrew Arnott
    @AArnott
    Awesome. Thanks, @neuecc !
    Aleksander Heintz
    @Alxandr
    Any chance of getting a new RC with the decreased immutable-collections? I'd love to get on the RC API to make sure it's still working for our application
    Andrew Arnott
    @AArnott
    Yes, I'll push one to nuget.org soon.
    Andrew Arnott
    @AArnott
    It looks like we'd better get #669 completed before the next release to avoid creating a nuget package id that will then be abandoned.
    Andrew Arnott
    @AArnott
    @neuecc It would be great if you could activate Azure Artifacts on your ils0086 account so we can create a CI feed for MessagePack, giving folks a way to always get the latest prerelease if they want it.
    Yoshifumi Kawai
    @neuecc
    @AArnott I've enabled Artifacts.
    Also, my v2 release blocker is gone.
    Andrew Arnott
    @AArnott
    Awesome
    Thanks. Do you mean you're willing to go for a 2.0 stable release, @neuecc?
    Andrew Arnott
    @AArnott
    @Alxandr 2.0.270-rc is now on nuget.org for your testing.
    Yoshifumi Kawai
    @neuecc
    @AArnott yes. (I've started to implement lz4 but it will be not in time to v2.)
    Andrew Arnott
    @AArnott
    ooh. sounds interesting. Thanks. I've branched off v2.0 from master. v2.0 builds a stable package and master is now tracking v2.1-alpha.
    I'm fine to give it a short while as 2.0-rc still to wait for feedback from @Alxandr or others in case there are any more changes we need to make.
    BTW as a matter of public API policy, I propose this, consistent with semver: API breaking changes get a major version bump. API additions, new features or noteworthy changes get a minor version bump. The only changes that can be made without a major or minor version bump are bug fixes.
    Yoshifumi Kawai
    @neuecc
    Okay. and agree to use semantic versioning.
    Yoshifumi Kawai
    @neuecc
    @AArnott sorry, is this final chance of breaking change? to implementing new lz4, I've noticed should change bool UseLZ4Comparresion to enum. #675
    Yoshifumi Kawai
    @neuecc
    thanks for the hard work in last few days.
    completely my concern is solved.
    wait the release:)
    Andrew Arnott
    @AArnott
    Yay. :)
    I'm happy with what we have now too. Since I don't need it stable this week, and we had some changes this past week, I suggest we give it another week of use to "stabilize" before removing the -rc and pushing the package.
    It's been great working with you, @neuecc