Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Alberto
    @agocorona

    Another addition is tmaskwich protect from asyncronous exceptions. The pattern is:

    do
        resource <-  tmask $ do
                    res <- openResource
                    onFinish $ \_ -> closeResource res
                    return res
    
       code that uses resource
       ....

    This is equivalent to bracket openResource closeResource $ code that uses resource but works for multithreading and any other effect.

    Alberto
    @agocorona

    We can could isolate this pattern as a primitive openClose:

    openClose :: (TransIO a) (a -> TransIO ()) -> TransIO a
    openClose open close= tmask $ do
                    res <- open
                    onFinish $ \_ -> close res
                    return res

    This is similar to the openClose that I defined time ago, but this one works for multiple threads.

    So. if open is a TransIO computation that return a "stream" of resources opened, they will be closed by onFinish when the three of threads generated after each one of them have been exhausted? It should work, but I haven´t checked it.
    Alberto
    @agocorona

    One advantage of the composability of transient is that any pattern is a function call wich is composable. Instead, bracket does not compose. I can open/close a resource in a thread and compose it with another open/close operation so that the continuation will work with both handles:

    do
        handle <- (abduce >> openClose open1 close1) <|> openClose open2 close2
        computationWith handle

    Since computationWith is not "sequestered" and managed by openClose. Instead, bracket, which is the openClose of IO, is a "framework" in the sense that captures your computation and execute it within his own environment. Later, it return a final value, but the process has been captured in the internals of bracket. That makes your captured computation second class.

    Alberto
    @agocorona
    Some breaktroughs in optimizing distributed computing with no external changes in the main primitives, but with the elimiantion of others like "fixRemote" which will be no longer necessary. Details soon. Less code, less primitives, faster. more flexible
    Alberto
    @agocorona
    New Serialization / cardano project
    Alberto
    @agocorona
    I have been busy integrating HTTP and Transient at the fundamental level. To do so I have to know when a stream ends, since usually HTTP streams finalize, while Transient streams from remote nodes had no implemented mechanism of notification that the stream has ended. I make use the mechanism recently implemented of detection of absence of active threads (finalizations, see above) to- send a SLast mesage to the closure that is waiting for more data in the other node. But this forced me to change a lot of things, since finalizations did not work with threes of threads where childs hang from parent threads and teleport, the basic primitive for sending messages did not honor that structure. Also messages among closures are now more optimized.
    Alberto
    @agocorona
    The mechanism of remote communication is as follows: when a program wants to communicate with the same program in another machine (that is when the primitive teleport is executed), the program store his continuation and identifies it with a hash number. The program looks for the last time it was called by that computer and the hash of that call. Then the program send the log of what has been calculated since then. That log is a path of serialized variables separated by "/". In the message there is the hash of the continuation that should continue.Upon reception the path is deserialized and assigned to the corresponding variables in the remote program thanks to "local" and "localIO" primitives. At this moment, both programs have the same state. Then the remote program continues normally and store the variables until another remote call and so on. If the program never has called the remote program, the log is sent from the beginning, to the continuation 0, which is the beginning of the program,the first statement after initNode
    Alberto
    @agocorona
    Let's say that a continuation is the result of the previous continuation that is in the path of execution plus the execution of the remaining code until another teleport is found. This continuation can be replicated in the remote node by sending to the last continuation known in the other node, the set of variables instantiated in the execution. the variables are serializez separated by "/" so that data interchanged looks like a path.
    Alberto
    @agocorona
    And this is the key to the next improvement on the interoperability of transient, which is necessary for my approved cardano proposal: A server platform that can implement smart contracts without ruptures of the flow so that the program can be easily readable, maintainable and verifiable by developers, users and maintainers. One of the key of smart contracts is that everything can be verified by the players.
    The hell of callbacks and web handlers are an hindrance for this purpose. As any modern multiplayer application, smart contracts are plagued by callbacks and handlers that break the flow and makes the specification flow to be unrecognizable in the code. The code of the server is called back from the cardano machine and from web clients.
    Alberto
    @agocorona

    Look at the implementation of a smart contract in Haskell, the guess game. Since cardano smart contracts are in the initial stages, plutus is a moving target. The current implementation differ in significant ways. Enter the plutus playground select the game example, which is the current implementation of the guess game.

    It involves two players Player 1 thinks of a secret word and uses its hash, and the game validator script, to lock some funds (the prize) in a pay-to-script transaction output. Player 2 guesses the word by attempting to spend the transaction output. If the guess is correct, the validator script releases the funds. If it isn't, the funds stay locked.

    Hardly you can see the flow of this specification in the code in either of the two implementations. And that is only the code that run in the cardano virtual machine. The interface with the user is generated by the simulator, but in a real application need his own set of handlers for web clients among other things.

    Moreover this is a very simple smart contract has only two players with one step each one. A contract may involve many roles (buyer, seller, arbitrator, evaluator etc) and many steps (milestones,time intervals, validations,payments). It should be necessary to express the program like the textual specification, as a sequence of steps.

    That problem is being tried to address by Marlowe. it is a high level DSL in which the contract flow can be expressed plainly without breaks of the flow. it can be analyzed statically so that some good properties could be guaranteed. It can then compiled to plutus in the future (see this paper), to create all the cardano endpoints necessary. The user interface is a task open for the DApp implementor. What the marlowe definition for the guess game would generate would be the plutus code that we see in the plutus playgroud.

    The Marlowe language however has not the flexibility of Haskell.

    That is how the game would look like using what I will develop. The program integrates the cardano endpoints "lock" "guess" and "startGame" defined above, although "startGame" seems not necessary in the current implementation. That is: it integrates the plutus code in a coherent workflow that integrates also the interactions with web clients:

    main= keep $  do
    
       amount <- webpoint "enter the amount to lock"      -- display the path/$d as parameter of the REST call from the type of the response
                                                          -- and the path.
                                                          -- also, it read it from the console if executed in console
                                                          -- generates {"msg"="enter the amount to lock","url"= "https://servername/e/f/w/$d"}
       gameId <- local $ inChain startGame 
       local $ inChain  $ lock amount
       newFlow  gameId                                    -- set a game identifier
       guessText <- webPoint "enter guess text"           -- generates{msg="enter guess text". "url"="https://servername/<gameId>/$d" }
                                                          -- and input it from console
       result <- inChain $ guess guessText                -- the inChain code of "guess" also pay if it is correct
       if result 
           then 
            removeFlow gameId
            webPoint "OK"
           else 
            webPoint "FAIL"
    
    from type to input: 
    class Input a where
        inputIt :: a -> TransIO a
    Alberto
    @agocorona
    Until the "newFlow" line, the program is executed by the initiator of the game. after that the program is executed by the one who guess.
    Alberto
    @agocorona

    There are some new primitives: webpoint inChain newFlow. The program can be executed as a console application and as a web server. as console application it display the the webpoints URLs and ask in the console for the fields asked for. In web mode, the users can send requests to these URLs. The responses are JSON registers that contains a message AND the URL of the continuation, with placeholders for the variables required and the type expected so that the flow will continue.

    The newFlow will create the identifier of the guess game that the first user has created when he locked the money in the blockchain with the previous execution of inChain startGame. The next webPoint will return the address of the flow created by the first player "https://servername/<gameId>/$d" where $d is the placeholder for the guess number. That is the continuation of the program. The first user then can publish that URL by other means to let users know that free money is available for them if they participate.

    when a webPoint is invoked the program run until the next webPoint. They are basically the teleport primitive that return a message besides his URL in a JSON register.

    When that URL is invoked with a number, the program will execute inChain guess guessText the guess code in the cardano blockchain. This code also pay the user in case the number is right. In this case, the flow is deleted, that means that any further invocation of the guess URL will result in a error.

    Many consideration for this code: It is multuser. There are code executed by different users as is required. the sequence of the contract is clearly seen. But there more subtle things: The execution of a typical smart contract could take months or years and requires only a few interactions, so it is not guaranteed that the program will run continuously and it would be a waste of resources to maintain all the context in memory for so much time waiting for a user input. Each webPoint is a continuation which works as a web endpoint waiting for some HTTP request. After a timeout, that continuation is saved to disk and removed from memory. When the request arrives, the continuation is restored in memory and executed.

    The code will run in console mode so that rapid testing of the program execution could be possible. The endpoint URLs are automatically generated.

    inChain is the react primitive with some additions to interact with the cardano blockchain.

    Alberto
    @agocorona
    I will call the library probably DAppFlow . To summarize, it integrates Plutus code generated manually or trough Marlowe, in a program that follow closely the smart contract specification and make it understantable and maintainable. It also check for more type coherence and less context data stored database since different users are involved in the same flow and values are stored in the stack of the continuation invoked.
    Alberto
    @agocorona
    Also, since it is a transient library, all the distributed multithreaded etc primitives of transient are available.
    Alberto
    @agocorona

    Plutus has a very sophisticated test system based on QuickCheck. However, it is complex . There are no intrinsic notion of sequence in quickCheck, so guess actions can be invoked before lock actions, two locks can come in sequence and hang the off-chain code because gameId has been coded as a global variable (Which is an easy temptation when there are no stack available and so on. these errors are spurious and does not appear in real usage scenarios, but forces the quckCheck programmer to add additional code to to emulate a logical test flow.

    In my proposal the very same program can perform tests by adding a third execution model the "test mode", bt injecting values in the webPoints and using plain assert lines to perform checks of consistency.

    The first is very easy since the TransIO monad is nondeterministic, so I can inject a set of values using choose or even random values. So that what is a single threaded program can perform a battery of tests.

    So webPoint woud do:

      webPoint = do
          if there is a connection open work as a HTTP endpoint
          else if test mode,  inject a battery of values, using  the QuickCheck.Arb instance  and `choose`
          else  input the values expected from the console using a Input class

    The rest is a matter of adding a assert and guard to verify properties and filter values for the tests.

    Since the flow of the program is the flow of the specification, there are no irrelevant test cases.

    Alberto
    @agocorona

    to inject a set of values instead of a single value, we can use the Arbitrary class of QuickCheck combined with the non-determinism of Transient:

           amount<-   threads 0 $ choose $ repeat 100  $ ungen arbitrary
           -- will inject 100 arbitrary values
           guessValue <-  threads 0 $ choose......
           guard condition   -- filter
           assert condition  -- check conditions

    That's the schema for testing. assert could use the same checks used in the quickCheck example, but the construction of sequencing by an ad-hoc state monad and the filtering of spurious sequences are not necessary.

    Alberto
    @agocorona

    This is possible because Transient has all the effects necessary. In this case, non determinism. The QuickCheck infrastructure need a few different monads: the spec monad and the dynamic logic monad. Besides the monads in which the production program will be coded.

    The first is an state monad that implement state transitions in the context of random sequences of in-chain actions (lock and guess in this example). Since this is not a natural usage case (lock should be used once at the beginning), the tester has to introduce auxiliary code to deal with spurious errors that are not present in the real contexts.

    The second, the DL monad, introduces nondeterminism to feed curated sequences of actions. This is a more realistic environment. However it only test the smart contract, not the whole logic of the program in which the smart contract may be embedded in.

    Besides the monads in which the production program will be coded, the programmer has emulated partially the execution flow two more times with these two monads.

    Instead, having the whole program expressed in a continuous flow allows for testing the smart contract with all the other logic together and do it once for production and testing, since the assert statements can be keept in production and the genetation of test cases can be a functionality of webPoint when run in test mode.

    Alberto
    @agocorona
    Writing this description clarifies my mind and solidifies my project ideas a lot.
    Alberto
    @agocorona
    since webPoint not only is a HTTP endpoint, since it can also input values from the console and inject random values for testing, it is necessary to rename it. Candidates are input interact io conversate ....
    input is a vintage name used in early languages. Looks hateful for functional programmers. I love it.
    Alberto
    @agocorona
    The effect of using this approach is a drastic reduction of the surface of the framework for developing and testing and verifying. I know that this produces an ambiguous feeling in developers. For one side, it makes things easier. For the other it leaves less programming things to invent and play with.
    So it is suited for practical people who want to get things done rather than Haskellers as such. But that is correct since this is intended for non-haskellers who want to create smart contracts.
    Alberto
    @agocorona
    Execution recover on demand
    As said above, to allow for computations whose lifespan is longer than the time between machine shutdowns, it is necessary to store and restore the execution state. I already had serialization of execution states in order to transfer them between nodes and I also had checkpoint and restoreto store in permanent storage the execution state, but I had to program the restoration of the execution state when a request arrives and to restore also the continuations that are watching from incoming requests in the execution path, including the one which waits for the current incoming request.
    Now I almost finished this functionality
    Alberto
    @agocorona
    Now that a network request to any continuation within the body of the program, I have to do the same with other kinds of events. Imagine a computation that watch for changes in the state of a smart contract and continue if a condition is met. That could run for an arbitrary amount of time; perhaps years.
    The standard way to deal with this is to create a computation mwatch :: IO (StreamData a) -> Cloud a similar to parallel but in the Cloud Monad, with the added capability of being activated when the program starts no matter where the mwatch sentectes are located in the code
    Alberto
    @agocorona
    As if the program had not been shutdown, except that all the irrelevant state is not restored.
    Alberto
    @agocorona
    "The standard way to deal with this is to create a computation " <- Obviously that is not the standard way. That is the cleanest way to do it. There is no standard way, since computations can not be serialized in Haskell. Probably a kind of callback handler which interprets a serializable EDSL could be a more standard way to do it
    Alberto
    @agocorona

    Serializable Haskell continuations

    The basic functionality of all that I want to do is to serialize haskell continuations incrementally so that from a cold start I can invoke any of these continuations. In this video I try to explain it:

    https://www.youtube.com/watch?v=MqmDxwa61fY

    
    data HI=HI deriving (Read,Show,Typeable)
    data HELLO=HELLO deriving (Read,Show,Typeable)
    data WORLD=WORLD deriving (Read,Show,Typeable)
    
    instance Loggable HI
    instance Loggable HELLO
    instance Loggable WORLD
    
    main= keep $  do
      flowAssign
      proc <|> restore <|> save
    
      where
      proc= do
        logged $ option "proc" "process"
    
        r <- logged $ return HELLO
        logged $ lprint r
    
        setc
    
        r <- logged $ return WORLD
        logged $ lprint r
    
        setc
    
        r <- logged $ return HI
        logged $ lprint r
    
      save= do
        option "save" "save execution state"
        liftIO $ syncCache
    
      restore= do
            option "res" "restore" 
            s    <- input (const True) "sesion >"
            clos <- input (const True) "closure >"
            noTrans $ restoreClosure s clos 
    
      lprint :: Show a => a -> TransIO ()
      lprint= liftIO . print
    
      setc =  do
        (lc,_) <-  setCont 0
        liftIO $ putStr  "0 ">> print (localClos lc)
    Alberto
    @agocorona
    Around this functionality I can restore, on-demand, the endpoint ls that invoke one of these closures and also I can restore the watching points that are waiting for state changes in the blockchain for an unlimited amount of time.
    Alberto
    @agocorona
    I had a earth attack when running a slightly modified case in which setc is inside a logged block and the program couldn't restore the state. Fortunately a simplification of the code of logged solved the problem
    Alberto
    @agocorona
    That is a frequent case: When the approach is sound, more requirements simplify the solution instead of complicating it, since they expose more generality which forces the coder to remove repetition that are basically the same algorithm applied to slightly different problems.Once one more case is added, it shows the repetition and the way to generalize it.
    Alberto
    @agocorona
    It is insane to recover the execution state, including stack frames, heap data, active endpoints, event listeners etc. it is like making a reusable rocket
    Alberto
    @agocorona
    Happily solved. A small bug that toke me a week
    When you enter in unknown waters it is impossible to know the deep of any problem. It could be the result of a corner case, an mistake, as is in this case, or could be something that could invalidate everything.
    Alberto
    @agocorona
    After a lot of time, I'm starting to see the light about how to integrate transient with cardano smart contracts
    Alberto
    @agocorona
    The problem is that the cardano contract uses freer-simple a effect library and uses it for a lot of different things. In particular effects are configured differently for testing with quickcheck, running simulations with the Plutus application backend (PAB) and finally attaching to a real Cardano node. The smart contract run isolated from the environment and even a direct plutus primitive can be reinterpreted by the PAB.
    practically everything in the contract becomes a request to the PAB, which in the current version uses STM for that communication. My idea was to attach directly the smart contract but I can not create an environment similar to the PAB since that is huge. So i would use as much of the PAB as possible.
    Alberto
    @agocorona

    Remote execution

    • RPC: identical binaries in both nodes, call the remote routine by his physical address: This is the classical RPC mechanism, implemented initially in C and implemented in Cloud Haskell. The invoked routine should be at the top level.
    • Send the code of the routine that will execute remotely, compile/interpret, invoke. Used in Unison. The invoked routine should be at the top leve (no scope).
    • Send the entire stack of execution (closure) to the remote node. Used in Scheme and other LISP related languages where running code can be serialized. Has the advantage of calling routines within any scope, so the code is not a set of unrelated routines. The code continues in the remote node without breaking the flow (composability). Has the disadvantage of the huge size of the closures to be serialized and transported.
    • Address the remote execution point by a path. Programs do not need to be identical, but to have the same structure. No need to transport code, can invoque routines within any scope. Invocation similar to a REST path, so it is lightweight. Instead of using absolute paths, a path from a known continuation point invoked previously can further reduce the data interchanged. Natural integration with HTTP-REST (almost for free). Since it composes all along the way and there are continuations, streaming is also natural

    -

    Alberto
    @agocorona
    After a lot of work studying Plutus internalities, I have managed to compile and run an example of a smart contract under the plutus application backend using the simulator. All of this under the IO monad instead of the complex effect system of the backend and Plutus itself. This is the fundamental step to run it under the TransIO monad
    Alberto
    @agocorona
    A lot of problems with versions of libraries since the plutus code is constantly changing
    Alberto
    @agocorona
    I was certainly a bit desperate until I saw the light one hour ago
    Alberto
    @agocorona

    It is sad the lack of interoperability of Haskell libraries because there is no agreed standard monadic stack. The effect libraries have some nice characteristics: what effect is allowed, but they introduce complex types, are slow (now) and need more boilerplate to run the effects. His composability is as bad as other models.

    Really, what a monadic stack need is pure and impure multistate for any data type, continuations and empty. That can be done within a Single and unique monadic stack. any number of readers and writers can be expressed in terms of state.

    Exception code is another problem. It should be first class instead of a kind of side code. Exception monads imitate the model of native exceptions, that everyone try to avoid. By putting exception code in a stack that becomes an state of the computation, an exception can invoke in reverse order the list of exceptions that are set for his type. if the handler fails "repairing" the exception, the next exception handler can be invoked. If it succeed, the associated continuation can continue execution at that point. That way exception flow becomes first class, because it resumes in the main flow, and inherit all the execution stack.

    Alberto
    @agocorona
    reminder talk about the dangers of "Extremely defensive programming"
    Alberto
    @agocorona

    Plutus

    The essence of what I have done with plutus smart contract is to run them trough the IO monad:

    main = do
      print "hello, I'm the IO monad!"
      initPAB simulatorHandlers
      cid <- runPAB $ Simulator.activateContract (Wallet 1) GuessGame 
      runPAB $  waitNSlots 3
    
      runPAB $ callEndpointOnInstance cid "lock" LockParams{secretWord="world", amount= Ada.adaValueOf 200}
      runPAB $ waitNSlots 3
    
      runPAB $ callEndpointOnInstance cid "guess" GuessParams{guessWord="world"}

    initPAB and runPAB are the primitives that I have developed to allow it.

    In this program I invoke the guess game in which someone stores money and a key in a address of he blockchain identified with cid . If another send to the same address the correct key, he get the money.

    The game smart contract is here

    There are three things here:

    • The smart contract, which run off-chain (the most part) and on-chain(basically, the redeeemer code).
    • There is also the PAB and the simulator. The PAB (Plutus Application backend) is basically a giant dependency injection stuff which establishes the effects and everything that is available when something is evaluated in the smart contract.
    • The third thing is the Simulator, which is the environment in this case, which the PAB bring to the smart contract. Other environment is the testnet which is a test blockchain and another is the real cardano blockchain.

    The program produces this output:

    Prelude Main> main
    [INFO] Slot 0: TxnValidate b135806142f3d020fc68e9e72fa6e8e7c8fce34b2a0f17fd23b633641f58944a
    [INFO] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Waiting for guess or lock endpoint..."
    [INFO] Initialising contract GuessGame with ID 9db4ce6e-479e-4fcf-977b-5d71c58d673c
    [INFO] Activated instance 9db4ce6e-479e-4fcf-977b-5d71c58d673c on W1
    [INFO] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Pay Value (Map [(,Map [(\"\",200000000)])]) to the script"
    [INFO] W1: Balancing an unbalanced transaction:
                 Tx:
                   Tx 29bd4df3f166bb1596ac1b7e57300d0fba664fde943b6ac45c682c366a482c0b:
                     {inputs:
                     collateral inputs:
                     outputs:
                       - Value (Map [(,Map [("",200000000)])]) addressed to
                         addressed to ScriptCredential: 96d9b249db40fe0b05dee4dfcecaafd3f18313e58f6054a7920608b8 (no staking credential)
                     mint: Value (Map [])
                     fee: Value (Map [])
                     mps:
                     signatures:
                     validity range: Interval {ivFrom = LowerBound NegInf True, ivTo = UpperBound PosInf True}
                     data:
                       "Hn\164b$\209\187O\182\128\243O|\154\217j\143$\236\136\190s\234\142Zle&\SO\156\184\167"}
                 Requires signatures:
                 Utxo index:
                 Validity range:
                   (-∞ , +∞)
    [INFO] W1: Finished balancing. 6b0e3b04f7b5c00104564c3d9bf8633d85d38bdf2e2c8d66a5357c18160bf0e1
    [INFO] W1: Submitting tx: 6b0e3b04f7b5c00104564c3d9bf8633d85d38bdf2e2c8d66a5357c18160bf0e1
    [INFO] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Waiting for guess or lock endpoint..."
    [INFO] Slot 4: TxnValidate 6b0e3b04f7b5c00104564c3d9bf8633d85d38bdf2e2c8d66a5357c18160bf0e1
    Nothing
    [INFO] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Waiting for script to have a UTxO of at least 1 lovelace"
    [WARNINGPrelude Main> ] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Correct secret word! Submitting the transaction"
    [INFO] 9db4ce6e-479e-4fcf-977b-5d71c58d673c: "Submitting transaction to guess the secret word"
    [INFO] W1: Balancing an unbalanced transaction:
                 Tx:
                   Tx bb08f5b021406223c4dab1fd3a99cd15fb191261a6a943bed747a96d8fe88000:
                     {inputs:
                        - 6b0e3b04f7b5c00104564c3d9bf8633d85d38bdf2e2c8d66a5357c18160bf0e1!1
                          "world"
                     collateral inputs:
                     outputs:
                     mint: Value (Map [])
                     fee: Value (Map [])
    Alberto
    @agocorona
    The smart contract communicates with the PAB trough endpoints, and some other low level primitives which actually are STM variables protected by a lot of type level stuff and effects. All of this is necessary for type safety and for isolating the SC from the environment, so that testing, evaluation etc can be done in different environments.
    Now that I can run it trough the IO monad, the smart contract execution can be intercálate with whatever I want. I can create a complete application with it.