Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Alberto
    @agocorona
    Indeed, now I have to update the registers already created in a subtle way that couldn't be predicted by the creators of the Fleet management application. The pattern recognizer would have helped me as well with that.
    Alberto
    @agocorona
    Alternatively, the application could have a record button, so that the sequence of actions could registered. At the end of the recording, the application could show the inputs entered as the first row of a textarea, so that the user can add more rows, that will be executed.
    Alberto
    @agocorona
    That is only possible because everithing in transient executes as a lineal sequence within the monad which can be recorded. even when the execution moves from the browser to the server and back while the user navigates and invoque actions in the server.
    Alberto
    @agocorona
    Another nice application using hplayground that I want to port to axiom (See this also)
    image.png
    run docker run -it -p 80:80 agocorona/tryhplay and ,point the browser to http://docker ip/try/mybudget.hs/edit
    Here is the tutorial about how this example application has been programmed : https://www.airpair.com/haskell-tutorial/intro-to-haskell-web-apps
    Alberto
    @agocorona

    Cool. Now I´m playing with a little modifications to make HTTP interfaces for Web programs much more meaningful.
    This program:

    main0=keep $ initNode $ do
        showURL
        local $ choose[1..3 :: Int]

    when executed runghc Test.hs -p start/localhost:8000

    produces this output which is the URL of the point at which showURL is located

    'http://localhost:8000/0/0/e/e/'

    opening the browser with this URL, the following is displayed in the browser:

    SMore/0/10003000/1/e/
    SMore/0/10003000/2/e/
    SMore/0/10003000/3/e/
    SLast/0/10003000/

    If you follow https://gist.github.com/agocorona/3b20c32d4a1fc47f48342a213c1c1fce you will understand: Each thread created by choose produces an output line, which contains all the logged by the program until the thread terminates. The last one, which closes the connection, is produced when there is no more thread running.

    Alberto
    @agocorona

    Any point in the cloud monad can be invoked by HTTP , then it executes the corresponding actions and receive all the responses, so the interoperability with any other language, system etc is guaranteed.

    Until now the connections were not closed when the response was complete. and to receive the response, the program should have an extra teleport. Now that is transparent and any cloud program receives proper responses.

    Alberto
    @agocorona

    The idea to blend all the kind of modalities of programming under a single execution model is a massive simplification which demand a accommodation of the mind. I though that it could be sooo good to have. The cloud monad allows so many things at the same time: It allows logging, stop and resume execution, distributed computing, web nodes, creation of HTTP endpoints... The transient monad under it allows for all the rest of the effects: threading, events, streaming, unlimited states, early termination, parallelism, concurrency, backtracking, compensations of transactions, better exceptions.

    All of this based on principle: continuations. It simplifies everything. We store continuations in the mind. When we have to stop a plan because we have to solve some details before doing the current step, we solve them and continue. That is the way the mind abstract from details and think at an high level: Because we use continuations.

    Alberto
    @agocorona

    True continuations remember the complete history of past steps, so the continuation can make use of any of the previous results. That happens with you and me when we execute a plan and stop the plan to do some other thing or some thing necessary for the continuation of the plan. That also happens in languages which support continuations as first class features.

    But look at callbacks and stackless continuations in general: They are like when you do a task and communicate to other the rest of the tasks written in paper, together with the data that you have obtained and think he would need. The person who continues is in another environment and is amnesic. If not all the data is available, he should ask you back or he would re-adquire the data from the environment or in some global state in which you stored more data.

    That is little composable. This means that, if the work need many interruptions to be continued by different amnesic people who continue your tasks, then such dispersion of data and processing and some possible bad global storage usage would make the whole chain of work very fragile. We find one of the shortcomings of bureaucracy and non-composable code.

    True continuations is as if you would continue each step after an interruption without forgetting anything. There is no loss of data and therefore there is no communication problem, no discontinuity no different environments.

    When the task to do is shared among many actors: persons or threads, then the problem of stackless continuations and callbacks are multiplied. Using true continuations is as if you can clone yourself with all the memory intact to do the different tasks. In programming you can execute the continuation in different threads. That is what transient does.

    Alberto
    @agocorona

    Continuations in human terms is the historical memory of what we did up to a point and the memory about the next steps to do. It is the key to think and act at a high level and also is the key to program at a high level: continuations in a way or another are used whenever the programmer need to invoke a lower level functionality and continue the program flow without being cluttered by these low level details: look at blocking system calls like file and stream operations, for example: they use continuations at the kernel level to resume your program when the file IO operation has finished. If you don't have these continuation abilities to call, then you have to handle the continuation manually, using callbacks, contexts or global variables with all the problems related.

    Even function calls are very lightweight continuations. They are the most lightweight continuations. They execute in the same stack context. The other side of the spectrum are the distributed continuations of transient, where a portion of the stack is serialized, transported, the continuation executed and the new components of the stack are returned back to the caller, which continues the execution.

    What is said about bureaucracy above can be mitigated by a close and true friendship. That makes the flow of information easy because in the chain of work the caller gives all information that the continuator need. They are all "a single person" . That's why unstructured working teams outperform theoretically well organized experts who work according to rules and procedures. But this hardly scales to big organizations or entire societies without authentic brotherhood among people who don't know each other. Christian brotherhood is necessary for an higher civilization.

    Alberto
    @agocorona

    If functional programming is programming with functions and functions are lightweight continuations then ... follow the reasoning.

    purity and type safety are poor selling points for functional programming that only scratch the surface of real programming problems

    Functions are to pure composition what continuations are to effectful composition. The latter does hit in the core of computing problems. Composing software modules that execute effects is what matters, since, either you like or not, effectful code is 90% of sofware programs

    Alberto
    @agocorona

    Continuations however had some problems. hard continuations using c primitives like setjmp and longjmp , that is are used by the OS ubiquitously to perform synchronous file IO and for hadling multitasking among other things. In these cases, the stack the instruction pointer , the IO info are stored and re-executed. Without that it wouldn't be possible to combine file IO operations with CPU bound operations. It wouldn't be possible multiuser and multiprocessing Operating Systems. That is, to begin with, a HUGE success in making pieces of code (programs) that can be developed independently and combine together to make bigger things. This agrees with some imperfect definition of COMPOSITION.

    At the language level, they have been used in languages like schema and Smalltalk for making asynchronous things synchronous, notably, continuation-based web frameworks and continuation based distributed computing. but continuation info store and retrieval demand time and space storage. When there are a lot of sessions the storage should be not in memory but in disk. These development tend to save entire suspensions, that includes data and code of the continuation which is huge.

    The Haskell continuation monad (or the codensity monad, which is a variation, exposes k, the continuation which is serializable in languages like the two above (generating big serialization chunks, as I said before), but it not in Haskell. it also has applicative, but no alternative composition. The instances do not consider the possibility that each member of the alternative, applicative or monadic combination run in different threads. As a consequence, if the continuation transformer includes the IO monad, only allows combination with blocking. whenever IO is executed within the continuation monad within an applicative, it blocks. This is not what would be necessary for composable concurrence.

    Alberto
    @agocorona
    Async-wait and stackless continuations (next topic)
    async-wait library in Haskell rely on continuations programmed at the runtime level, since it uses TVars. async-await in other languages are callbacks disposed in cool ways. They are stackless: the continuations have no access to the stack of the caller, that is, the intermediate result above in the execution flow therefore they are not composable as functions and monads are.
    Alberto
    @agocorona

    To summarize, there is no way to express algebraic combinations of parallel and concurrent pieces of software so that these pieces make use of past results in scope as if we were running a single thread. There is no way to combine then algebraically with different asyncronous inputs (streams, sockets, mouse events, keyboard input). Much less when all of this run distributed among different nodes.

    Transient does it.

    Alberto
    @agocorona
    There are other problems related with continuations: Unlike functions, which use the same stack than the caller, continuations should store the context and recover it. This includes in some way or another, a heap, a stack or a closure name it as you like and the rest of the code to execute which may be indicated by a pointer. This takes space. A lot if there are many of them. But threading, multitasking, file IO... almost all OS features use this technique and no programmer would get rid of these services to program in the bare metal to gain some performance and inprove the memory usage.
    Alberto
    @agocorona

    However these continuation-based services are created in the 70s. The new programming problems of the 90s: Web programming and mouse input or in general, concurrent management of events was not solved the same way. While the OS brough primitives like epoll which partially solved the problem of composition of inputs, in the user space, the frameworks for web programming , distributed computing (actor model) and GUI programming opted for callback-based schemas. Which is a hell.

    The barrier to the adoption of composable frameworks for these problems has been the difficulty to make continuations compose well among them at the language level unless you block. For example if I want to read two files (which use continuations at the OS level) in the program either you do it sequentially or you must fork another thread, which break composition. In transient, you can express this algebraically with (async (read file1) <|> async (read file2) >>= process and each file will be processed in parallel. If you use <> instead of <|> the two files will be concatenated and the whole content will be processed in one thread. The same construction could be used for sources of events: ((react onmouseEvent >> return "hello")<|> async(read file) ) >>= process and these expressions can be combined as well with the same operators.

    Alberto
    @agocorona
    To extend the benefits of continuations to remote execution and distributed systems, the continuation context should be replicated in the remote computer and the same operation should be done when the result comes back to the local computer. If both computer share the same code, basically, the execution stack and a pointer should be transported back and forth. That is huge. However with some techniques it can be reduced greatly. For example instead of all the stack, only transport the values which are in scope of the cloud monad, and not other intermediate values that would be out of scope. Also, by incremetally transporting and reconstructing the stack back and forth, so when the called node respond to the caller, instead of transporting back the whole stack, it transports what changed on it.
    Alberto
    @agocorona

    The stack can be transported as a path of serialized variables separated by "/". this makes such continuations human readable like HTTP GET paths.

    main= keep $ initNode $ inputNodes <|> do 
        let name= "WORLD"
        node <- local $ do
                    ns <- getNodes
                    guard (length ns > 1) <|> error "I need to know the other node"
                    return $ ns !! 1
        resp <- runAt node $ do
            localIO $ print "producing response here"
            return $ "hello "++ name
        localIO $ print resp

    In the program above, the message sent to the remote node in runAt is:

    SMore/0/30105000/e/f/w/("localhost",8001,[])/e/

    And the response of the remote node is

    SMore/30105000/40106000/"hello WORLD"/e/
    SLast/30105000/40106000/

    The first means: I'm the continuation 30105000. I call you at the beginning of your program (continuation 0) please consume these variables and continue executing. Note that the list of nodes from getNodes are out of scope and not transported.

    The latter means: I'm the continuation 40106000 and I call back your continuation 30105000 which is the one who called me. Please add "hello WORLD" to the stack of that continuation (which is the value of the variableresp) and continue executing. There "SLast..." response means: I have nothing to send from 40106000 . you can DELETE your stack context to save space. I deleted mine.

    The latter is a solution for another problem: how to free continuation contexts when there are hundreds, thounsands of them.

    The fascinating thing is that the remote execution could be invoked by a simple HTTP client. With curl, for example, I can invoke any local statement of the program, and receive his response.

    if the second node is at localhost:8001:

    >curl http://localhost:8001/0/0/e/f/w/("localhost",8001,[])/e/
    SMore/30105000/40106000/"hello WORLD"/e/
    SLast/30105000/40106000/
    Alberto
    @agocorona
    Sorry if you don't understand all that happens in the example above and as always I'm so bad explaining myself in English. I want to create some videos.
    Hugh JF Chen
    @hughjfchen
    @agocorona Any update of the transient library which incorporates these ideas? Or it's available already in the release?
    Alberto
    @agocorona
    hi @hughjfchen that works already in the https://github.com/transient-haskell/transient-stack repo. There are some bugs and adjustments that I did not yet upload but there is it. I will upload the fixes as soon as I can
    Alberto
    @agocorona

    Streaming
    Why SMore and SLast? The response in the above example has been received. Why wait for SLast to close all the contexts? Because unlike all RPC mechanism the continuation based remote communication support streaming and keep being composable. There is no loops and no callbacks. There are no special construct, no new operators, just some more primitives. The responses could be from 0 any number, including infinite.

    This

    resp <- runAt node $ do
            i <- choose [1..5]  -- better: threads 0 $ choose [1..5]
            localIO $ print "producing response here"
            return $ "hello "++ name ++  show i

    Will produce five SMore responses and the SLast message. Better prefix choose with thread 0 otherwise five threads will produce simultaneously their responses and compete to write in the network socket to send his responses. with threads 0 the main thread will execute a loop and will write the responses one by one. Network streaming solved and a number of things like thread parallelism are involved in this and complement themselves nicely if this it is necessary.

    choose is a combination of async composed with the alternative operator which tries to use all the threads possible.

    Asynchronous communications
    It also support 0 responses, in which case, SLast will be sent immediately:

    runAt node $ do
            localIO $ print "no response produced here"
            empty
    Alberto
    @agocorona

    Generators
    Generators in python can yield lists of values that can be used as ranges in for loops. Using generators, the fibonacci serie is more or less:

    def fibonacci():
         yield 0;
         a=0
         b=1
         while true
             yield b
             a,b= b,a+b
    
     for num in fibonacci
       print num 
       if num > 10000 break

    Yield can be called at any point of the execution of the generator. In the example is used in two locations. nextcan be used to iterate the values of the generator. This is something that can be implemented using continuations but it seems to me that this kind of coding is not right. It is also imperative and use loops explicitly. It is however example of pull like streaming. Normally imperative languages are push-like since execution is from top to bottom. Pure Haskell code is lazy of course, so it is pull like. Monadic code is also push-like.

    A transient program can return more than one value. Here the function fibonacci return a value b and a infinite recursive set of values. if no thread limit is established it will try to return all of them at the same time using infinite threads. It is a more functional and recursive definition of the iterator, without using mutation. There are no loops.
    killBranch kill the thread/s that does the work the hard way. I need one more civilized way to end a thread since kill is asynchronous and takes some time to make it work. Probably endBranck

    sequential= threads 0
    yield= async . return
    
    mainfib= keep $ sequential $ do
       let fibonacci= fib 0 1 
            where
            fib a b= yield b <|> fib  b (a+b)
       num <- fibonacci
       liftIO $ print num
       when (n > 10000) killBranch
    Alberto
    @agocorona

    An generator is like a coroutine , a cooperative thread which yields a result and stop. next continue execution of the generator until the next yield. Generators, like iterators, do not compose. I can not write for num in fibonacci and factorial. but I can write in Transient: num <- fibonacci <|> evens

    Since it is pull based, it is passive, I can not inject events from outside to the loop. For example, I can not read from the keyboard, except by blocking and this is not good for composition, although generators do not compose, as I said. In the other side, I can write in transient: num <- fibonacci <|> input (< 10000) "enter a number" . Both sources will be injectected in the computation.

    Alberto
    @agocorona
    Even I can write num <- fibonacci + evens and will generate a single serie which will be the sum of the terms of both.
    evens = evens' 1
       where 
       evens' x= yield x <|> evens' (x+2) 
    
    yield= async . return
    Alberto
    @agocorona
    where I wrote above "factorial" i meant "evens"
    Alberto
    @agocorona
    Also, the terms of any expression could be distributed among different nodes applying local and runAt node . The more I look at the composability of transient , the more I get amazed.
    Alberto
    @agocorona
    image.png
    Why Continuations are Coming to Java https://www.youtube.com/watch?v=9vupFNsND6o .
    They implement generators from continuations
    2 replies
    Alberto
    @agocorona
    The bad thing with these special constructions is that the code looks second class. The generator is scheduled by another code, the one which contains the for next loop that is first class. It is as if we go back in time to non-preemptive multitasking like in the early days of Apple and Windows. It is curious and weird so It is from my point of view not the best example of what can be done with continuations, and that is the cause of the reputation of continuations as sources of confusion instead of clarity.
    Alberto
    @agocorona
    Similar considerations can be applied to many streaming libraries in Haskell. They use their own kind of scheduling, have special operators and define special sections such "sources" and "sinks". That is a kind of second class code. I can not compose sources. I can not inject events . I can not do general programming in the middle of a pipe or conduit chain. I can only use special primitives specifically made for it. All is specialized. As a consequence they tend to be huge frameworks.
    Alberto
    @agocorona
    They are increibly ingenuous and sophisticated. At his core they use continuations to create special execution models, which is rather difficult to manage and need his own scheduling mechanism trough all the pipe. Transient in the other side only add some features to the IO execution model. implicit looping and threading in this case. But, for the most part, the pieces of the puzzle are not constrained to any particular framework with particular combinators. It is ordinary haskell code.
    Alberto
    @agocorona
    I understand that conduits and pipes and all the streaming packages in Haskell are Domain-Specific Languages (DSLs) for streaming and all these restrictions makes sense if we consider them as such. BUT streaming is a particular case of a more general kind of computing which includes non-determinism, parallelism, folding generalized to concurrency, events.. in general, processing and returning not one but a non determinated list of values, from zero to an infinity of them. And that kind of computing is the all-encompassing computing. It is general computing, for which all the others are particular cases. So if you want to deal with all these cases, you should enhance the default computation model, not to restrict it with a new particular one. And your library should be general purpose, not a DSL.
    1 reply
    Alberto
    @agocorona

    What I mean as folding generalized to concurrency:

    main= keep' $ (foldr (+) 0 $ map yield [1..10::Int]) >>= liftIO . print
       where
       yield= async . return

    it generates a 10 numbers in 10 threads and concurrently sum them.
    By the way, choose= map . yield

    Alberto
    @agocorona
    results in general can come from different threads so concurrency is a particular form of a general concept of folding. Note that i use a standard fold primitive foldr.
    Alberto
    @agocorona
    Imagine that instead of natural numbers, they are cryptocurrencies mined in different nodes and they are received by the initiating node of the distributed computation. It is a example of parallel processing. There is no concurrency there.
    main= keep $ initNode $ inputNodes <|> do
               local $ option "go"   "start mining"
               nodes <- local getNodes
               curr <-  foldr (<|>) mempty $ map (\n -> runAt n  minecurrency) nodes --  ==clustered minecurrency
               localIO $ print ("new currency mined:",curr)
    
    minieone :: IO Currency 
    
    minecurrency= local $ waitEvents mineone  :: Cloud Currency
    Alberto
    @agocorona
    changing the style by avoiding "do" and using a more streamy look may make it more fancy and cool but I don´t believe that putting everything in one-line add anything to streaming.
    Alberto
    @agocorona

    Example loops:
    for loop:

    forM [1..10] $ \x -> proc c
    next

    Equivalent to

    mapM  proc [1..10]
    next

    Executes the computation proc for each one of the values sequentially. For example:

    formM ["file1","file2"...] $ \fil -> do
           content <- readFile fil
           process content

    If we need to do it in parallel for each value of the loop

    fork $ choose[1..10] >>= proc    --      fork x=  (abduce >> x) <|> return()     
    next

    Use threads to reduce the parallelism.
    This executes the loop in another set of threads. The current one executes immediately next

    If we want to wait and pass the list of results of the loop to next:

    results <-collect 0 $ choose[1..10] >>= proc
    next results

    collect 0 wait until no thread activity is present in his argument and gather all the results

    Alberto
    @agocorona
    Since proc is now a transient computation, proc can return (yield) any quantity of results, depending on how you define it, so, in general, the length of the list gathered by collect is not the length of the list of choose. If it is sure that proc will return just one result, then group instead of collect would do the job and it is faster.
    Alberto
    @agocorona

    Resource management (again)
    I have a problem with resource management when the resource is shared among different threads. Imagine a huge file that could be processed in chunks by different threads, but also a list of results which are being filled by different threads or even different distributed computations. Coding each particular case is easy, but not the general one. It is necessary to detect the extinction of thread activity so that the resource which is used before the final action: closing a file, returning a result etc.

    Until now I did it in some cases by catching runtime errors in MVars shared among threads, but the detection of MVar readers with no writers active is very fragile and may not work when the code is relocated. I could use TVars but I would like to use neither of both in order to make transient as simple and portable as possible. I also use a periodic inspection (polling) of continuation states to know when to send a SLast message to a waiting node in order to discard a continuation for which no node will call back. All of this could be substituted by a single and faster mechanism.

    It is difficult because, in the general case, threads can spawn more threads. Moreover, a single thread can return many results (or no result). So simply counting the number of results to decide when it is finished is not possible.
    Finally I managed to improve onFinish for this purpose, even if there are many threads involved. Now a onFinish will only close the resource when there is no thread activity after it.

    This code spawn three threads by choose. I labeled them using labelState so that they could be identified better with showThreads. At the end of the execution of the three threads the finish message is printed. Provisionally the Finish message have the identifier of the last thread which finalized his execution:

    main=   keep' $ do
        onFinish1 $ \r ->  liftIO $ print ("finish",r)
        i <-  choose[1..3]
        labelState $ fromString $ "thread from "++ show i
        th <- liftIO $ myThreadId
        topState >>= showThreads
        liftIO $ print ("end",th)

    The output is:

    tests# runghc   -w -i../src Test
    ---------Threads-----------
    top 76
      top 77
        work,thread from 1 78 <--
    ("end",ThreadId 78)
    ---------Threads-----------
    top 76
      top 77
        work,thread from 1 78 dead
        work,thread from 2 79 <--
    ("end",ThreadId 79)
    ---------Threads-----------
    top 76
      top,thread from 3 77 <--
        work,thread from 1 78 dead
        work,thread from 2 79 dead
    ("end",ThreadId 77)
    ("finish","ThreadId 77")
    
    tests#
    Alberto
    @agocorona
    dead means that the thread has finished and his register is being eliminated. <-- is the thread where showThreads is executed
    Alberto
    @agocorona

    In the same way I've created a binary onFinish which will return a result when all the threads of the first operand terminates so I can return a result. The schema is:

    main=   keep' $ do
        container <-  create some mutable container to gather results
        results <- SomeProcessWith container `onFinish1'` \_ -> return container

    This has direct applicability to collect that until now worked with a fragile detection of MVar exceptions and the exit of `keep'' which worked the same way. The former unary onFinish has applicability for sending SLast message to remote nodes in order to make them garbage collect their continuations.

    They work using the backtracking mechanism. There may be different onFinish along the execution of the monad wich will be executed in "last in - first out" order, if there are no activity under them.
    Alberto
    @agocorona

    With this solution, the scope in which the resource is visible match the lifespan of the resource. It is not needed to bracketing the use of the resource by inverting the control with withResource. For example:

    test=  do
      result <-  do
        onFinish1  $ \e -> liftIO $ print ("END", e) 
    
        liftIO $ threadDelay 1000000
        return "something"
      liftIO $ print result

    seems that if result would be a file descriptor, it would escape the finalization, but that is not the case since the finalizer works for all the execution flow which passed troug onFinish

    > runghc test
    "something"
    ("END","ThreadId 80")
    Alberto
    @agocorona

    Going all the time trough unexplored territories implies expecting unexpected difficulties and setbacks. It takes time before problems are solved. Initially with solutions a little messy under the hood. takes a lot of tests of alternatives and factorization to reach the point where there is no better solution.

    Better solution= the smaller set of components that compose.

    Alberto
    @agocorona
    But it is rewarding when I can build management of resources as a fine layer upon primitives for backtracking and thread management. And the latter, over primitives for continuations.
    Alberto
    @agocorona
    Endpoints
    I'm experimenting with a endpoint primitive that will make the management of any transient program trough HTTP REST more easy and natural