Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    jongoldDB
    @jongoldDB
    Just to clarify, if you use async calls, concurrency can happen, but not parallel execution, since everything is executed in a single threaded manner (like Javascript). Now you can get around this of course if your in C#, but we don't recommend it...
    MedAnd
    @MedAnd

    Hi @jongoldDB - safe to say I have a way to go before understanding Ambrosia, and will circle back to above API question at some stage.... but already excited by the potential :relaxed: For example from the docs it seems multi producer is already supported via the buffer: we log all incoming requests (which can come from multiple sources)?

    On the consumer side though it seems that expensive partitionable workloads need to be sharded, where each shard runs on its own set of cores... wondering if this means even on the same VM/container and if code examples exist for this scenario... how should long running operations (calls to restful APIs, databases, indexing engines for example) be reasoned about & integrated with?

    MedAnd
    @MedAnd
    Friendly ping @jongoldDB - anything you could share on the Service Fabric examples... also any update on the project road-map, seems to have gone a little quiet 😊 Thanks!
    jongoldDB
    @jongoldDB
    Hi MedAnd. We are close to rolling out a version which allows deployment of both processes into a single process, which is what's needed for service fabric support and samples. We should be rolling it out with samples sometime over the next month or so. In terms of our longer term roadmap, the next big features involve support for device oriented scenarios, and elastic scaleout, both of which will be worked on intensively over the summer. Let me know if you have further questions, or a desire to contribute :)
    Also, we are working on a new version of the whitepaper, which will contain more detail about the architecture of the immortal coordinator, if you're curious.
    MedAnd
    @MedAnd
    Thanks for the update @jongoldDB and nice to hear the project is full steam ahead! The big features that are being worked on sound like they will make this project more developer & production friendly... Can I assume you have internal Microsoft projects that will validate the elastic scale-out approach, performance etc? Re: contributing... I hope so at some stage :relaxed:
    jongoldDB
    @jongoldDB
    Hi @MedAnd , the new whitepaper is now available through the old link. There are quite alot of changes and additions, and I think things are much more clear. We currently have one big internal customer, which is using Ambrosia as it is. We just recently got another, which may end up relying on our elastic scaleout mechanism, but it's hard to say at this point. There is alot of work to be done to light up this feature, and being researchers, our performance ambition is unprecedented in the actor space, which is what makes this research :)
    Christopher S. Meiklejohn
    @cmeiklejohn
    in the HelloWorld example in the Ambrosia repository it says the following:
    "To run HelloWorld you'll also need the Ambrosia tools that are distributed in compressed folder. To get these, Download the compressed folder Ambrosia-win.zip"
    But, I can't seem to find where that Ambrosia-win.zip file is located to download?
    Christopher S. Meiklejohn
    @cmeiklejohn
    ah, this is a release
    Christopher S. Meiklejohn
    @cmeiklejohn
    Anyone have any thoughts on having Ambrosia crash with this error?
    Disabling dynamic assembly loading
    Enabling sideload for vertex: ambrosia (Ambrosia.AmbrosiaRuntime)
    Ready ...
    Logs directory: C:\logs\
    Reading a checkpoint 1257 bytes
    Attaching to server
    FATAL ERROR 4: Error in local listener data stream:System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
    --- End of inner exception stack trace ---
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
    at System.IO.Stream.ReadByte()
    at Ambrosia.StreamCommunicator.ReadInt(Stream stream, Byte[] buffer) in D:\a\1\s\Ambrosia\Ambrosia\StreamCommunicator.cs:line 414
    at Ambrosia.FlexReadBuffer.Deserialize(Stream S, FlexReadBuffer flexBuf) in D:\a\1\s\Ambrosia\Ambrosia\StreamCommunicator.cs:line 145
    at Ambrosia.AmbrosiaRuntime.LocalListener() in D:\a\1\s\Ambrosia\Ambrosia\Program.cs:line 2744
    KILLING WORKER:
    Christopher S. Meiklejohn
    @cmeiklejohn
    I seemed to have found the issue.
    Chris Lovett
    @lovettchris
    Oh, what was the fix ? I'm hitting the same thing in the HelloWorld sample... after it was working fine for a while...
    jongoldDB
    @jongoldDB
    I would need to see the error in the ImmortalCoordinator window to be sure, but I'm guessing you want to start over and didn't remove the directory the logs are stored in to reset things
    We use the presence of the log directory to indicate whether we should recover from logs as opposed to starting from scratch.
    If the log files are missing but the log directory is still there, Ambrosia will assume you intended to recover, and when the log file and checkpoint files are missing, will kill the ImmortalCoordinator with a missing checkpoint error
    jongoldDB
    @jongoldDB
    Wait. I just realized the output is from the IC window :)
    How are you running it? Is it an active/active scenario?
    This error means that the server process ended. What does the server process window say? Is there an exception?
    Chris Lovett
    @lovettchris
    ignore my comment, deleting the logs fixed my problem. I swear I tried it before, but must have missed something. Works again now :-)
    MedAnd
    @MedAnd
    MedAnd
    @MedAnd
    Friendly ping @jongoldDB - channel has been
    a little quiet... any news or updates? Thx.
    jongoldDB
    @jongoldDB
    Hi @MedAnd, sorry for the quiet time, we have a bunch of backlogged updates which are sitting behind a CRA update, which has been very arduous. I think we'll have the version out with the CRA update around the beginning of next week. After that, we'll have another update for the backlog. Thanks for the patience :)
    MedAnd
    @MedAnd
    No worries & look forward to reading what the team has been working on. Maybe out of scope but any thoughts on the new dapr.io announcement?
    Bartolomeus-649
    @Bartolomeus-649
    Just looked at the HelloWorld sample and read the "HelloWorld Explained" document (https://github.com/microsoft/AMBROSIA/blob/master/Samples/HelloWorld/HelloWorldExplained.md).
    Since the async approach, where you get the result of the calls to the Server, is problematic, what is the recommended method for returning data when calling a remote method?
    MedAnd
    @MedAnd
    Hi @jongoldDB, working with Azure Functions recently I posted the following idea & interested in your thoughts πŸ™‚ https://github.com/Azure/azure-functions-durable-extension/issues/1302#issuecomment-611187573
    jongoldDB
    @jongoldDB
    Hi @MedAnd, I saw the conversation on durable functions. Sebastian was actually one of the initial contributers to Ambrosia, and, as he indicated, there is some thinking around using Ambrosia there. Like Ambrosia, durable functions DOES have virtual resiliency, although the workflow scenarios they target don't require the high performance we are targeting in Ambrosia. Still, this doesn't mean they couldn't do so with a sufficiently performant back-end :)
    jongoldDB
    @jongoldDB
    Hi @Bartolomeus-649, since serializing and deserializing the context of an awaiting task is pretty experimental, for any sort of service, which runs for a while (as opposed to a job, which runs for a short period of time), and will likely experience upgrades, patches, etc..., we advocate a more message based way of communicating. So in essence, a return value is sent back to the call originator as a separate message/RPC. You could add some sort of an ID to the initial call and the return value RPC which would allow you to connect the return value to the initial call, but it would be your responsibility to squirrel away whatever information you need to complete the action upon receipt of the response on the client. Essentially, you're implementing your own continuation. This would also allow you to evolve your code in a client upgrade scenario by explicitly providing transformations during upgrade of the continuation state, as well as new code to handle the continuation itself as part of the upgrade. In practical terms, this is probably the best you can do in terms of code evolution. Make sense?
    Oscar Ruckdeschel
    @ibiBgOR
    Hey there. I just started with AMBROSIA (and C# to be honest) for my masters degree. Currently I'm extending the Streaming Example. Here I externalized an Immortal for the language (so not only relying on the language given from twitter but on a trained machine learning algorithm). Also I externalized an Immortal for the Sentiment.
    While I'm developing I notice that the awaiting of the RPCs do not really work as well as i imagined. But my main problem currently is that the ImpulseHandling mechanism sometimes just don't work. Like no message is sent to the Impulse handling method. Furthermore this is not only a problem for this edited code but also for the HelloWorld example project (Client 2). I have no clue on why this happens "randomly" out of the blue. Is there any chance you can help me?
    Btw. I just started "rewriting" the code away from awaiting the answers of the RPCs and using messages and message handling via IDs by my own, as supposed by jongoldDB!
    Oscar Ruckdeschel
    @ibiBgOR
    Oh. I just deleted "everything" starting from unregistering all immortals and ending up manually deleting the blobs and removing the tables in azure and my impulses seams to work again. What could be the reason for this problem to occur? Is it possible that multiple "RegisterImmortal" calls to azure are bad? Or does restarting Immortals after deleting log files (and folders) result in this strange behavior (as I can reproduce this error with the second run of my immortals).
    MedAnd
    @MedAnd
    maybe also a good opportunity to ask @jongoldDB for a general project & roadmap update... been very quiet here for many months πŸ™‚
    Rahee Ghosh
    @raghoshMSFT
    @MedAnd thanks for checking in! Jonathan and the rest of our team are very close to having a new release out with some new features for client-oriented use cases in the next week or so. We'll share more information when that is out and look forward to feedback from folks here :)
    jongoldDB
    @jongoldDB
    Hi @MedAnd, we just released a new version of Ambrosia with a number of new, client oriented features. Take a look at the release notes, and new documentation, and let me know if you have questions :) In terms of a roadmap, part of the quietness has come from what we view as an exciting opportunity to use Ambrosia at the edge. There is still lots more to do, and coming up, but the general idea is to virtualize client software, similar to the way we've done this for server oriented software. Since we are targeting things like cell phones, there are many assumptions about the environment which work in the datacenter, but not at the edge, like the ability to mount network file systems, and the ability to spin up processes easily. These issues, and more, have been addressed in this release, and you'll see more in the upcoming releases. Stay tuned :)
    jongoldDB
    @jongoldDB
    Hi @ibiBgOR, sorry you had difficulty. If you encounter a problem like this again, and/or have a repro, please let me know. Also, if you have questions about Ambrosia, or want to tell me about your project, I'd love to know how you're using it in your masters' program.
    MedAnd
    @MedAnd
    Hi @jongoldDB - any plans for AMBROSIA to integrate with or power Azure Functions / Durable Functions πŸ™‚ ?
    Bartolomeus-649
    @Bartolomeus-649

    @jongoldDB

    ... since serializing and deserializing the context of an awaiting task is pretty experimental, for any sort of service, which runs for a while (as opposed to a job, which runs for a short period of time), and will likely experience upgrades, patches, etc..., we advocate a more message based way of communicating. So in essence, a return value is sent back to the call originator as a separate message/RPC. You could add some sort of an ID to the initial call and the return value RPC which would allow you to connect the return value to the initial call, but it would be your responsibility to squirrel away whatever information you need to complete the action upon receipt of the response on the client. Essentially, you're implementing your own continuation. This would also allow you to evolve your code in a client upgrade scenario by explicitly providing transformations during upgrade of the continuation state, as well as new code to handle the continuation itself as part of the upgrade. In practical terms, this is probably the best you can do in terms of code evolution. Make sense?

    Nope, does not make sense at all

    1. If you know of a way to make calls and get a reply back, then include it in the framework.
    2. Since it is rather common that you want to get back a response from functions you call, every developer will need to invent a method to do this by themselves. And this is non-productive code, code that only exist to be able to get the value back that they wanted in the first place.
    3. When basic functionality like getting a value back from a call is missing, it affect the productivity and make the platform expensive to use.
    4. It goes against some of the fundamental design goals of both C# and dotNET, where to goal is to write as few lines of code as possible (within reason), since code contain bugs and require maintenance, and all code should be "productive code", that is, code should exist to solve the problem at hand, not just be there to make the real productive code work/run, also known as plumbing.

    So, exactly why should every single developer need to invent a method to get a value back by themselves?

    jongoldDB
    @jongoldDB

    Hi @Bartolomeus-649,

    Thanks for your thoughts around this difficult issue :) Since we can no longer support the serialization of Task state in an even experimental way, we will soon be releasing, with Ambrosia, helper libraries that make it easier to author calls which are expected to generate responses, making it much easier to write this sort of code. Note that some of the issues associated with handling upgrades still remain, and are endemic to distributed systems which are long lived, and have evolving APIs. That said, we will keep this in mind when designing these helper libraries, and will hopefully do better in this regard than the fully automatic, but brittle, outcome you get using the old experimental Task serialization approach.

    To add some more clarity around this decision, unfortunately, C# just doesn't give us the hook we need to serialize and recover application state that includes Task state. Without this, there is no recourse we know of other than something experimental which makes assumptions about the C# runtime which have already broken. As a result, we can no longer support even the experimental approach we had running before (see the latest version). Note that if C# included this capability, we could immediately make use of it. Also, other languages which support recoverable async state can be supported by Ambrosia with an appropriate language binding, although this is asking a lot of today's languages and runtimes, and I'm not sure if such a language exists. We look forward to your feedback on our future release which addresses these issues for today’s C#.

    MedAnd
    @MedAnd

    Hi @jongoldDB - just wanted to share a potentially complementary project has gone live from Bart J.F. De Smet (of Rx fame). Interested in thoughts on how Ambrosia compares, would it make sense to collaborate as both are MS birthed projects πŸ™‚ etc?

    https://reaqtive.net/

    Ghost
    @ghost~5ecdd057d73408ce4fe4fd1b
    Greetings :)
    I am just discovering Ambrosia.
    Suspect it will be an integral part of BrightChain... which is why I'm here... you're all like-minded people working in a similar area, can you help launch this thing into the stratusphere?
    checking out reaqtive myself now...
    I am trying to get BrightChain into MS Research
    Ghost
    @ghost~5ecdd057d73408ce4fe4fd1b
    I'm one relatively limited developer and it will take me an eon to build the vision i have which I think so far others seem to think it's generally a good plan.
    Ghost
    @ghost~5ecdd057d73408ce4fe4fd1b
    Is there a gitter / discord for FasterKV?
    Reuben Bond
    @ReubenBond
    Ghost
    @ghost~5ecdd057d73408ce4fe4fd1b
    sweet!
    thanks!