Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Hugh JF Chen

    One of the main problems in transient is to know if block of code continues or not, since the execution can be stopped at any moment. Moreover, since the transient is inherently multithreaded and single threading is just a particular case, some threads can follow a path in which they stop and other don´t.

    I must admit that this is the most confusing problem I'd been facing since the first time I met transient. So a type level safety is really welcome and necessary

    I suspect that if-then-else expressions with monadic code will no type-match in many cases when using this library.

    For example:

    test10 x= if x == 5 then liftIO $ print x else askfornumber
    askfornumber= do
          n <- input (<10) "give me a number >"  
          liftIO $ print (n :: Int)

    The error produced at the else :

    src/Transient/TypeLevel/test.hs:41:48: error:
        • Couldn't match type ‘Terminal’ with ‘IOEff’
          Expected type: T '[] '[IOEff] ()
            Actual type: T '[] '[Terminal, IOEff] ()
        • In the expression: askfornumber
          In the expression:...

    because the "then" side has type T '[] '[Terminal] () since it uses terminal input, while the "else" side uses IO. Since both branches should have the same type, hence the error.

    To solve the problem, the best way I found until now is to use guard in combination with alternative, which is IMHO, a good style that improves readability:

    test11 x=  is5 x <|> askfornumber
     is5 x=  do guard $ x==5 ; liftIO $ print x

    This type check. What is the signature? A mess of Termination-maybe-non-termination?
    Using GHCI:

    *Main> :t test11
      :: (Eq a, Num a, Show a) => a -> T '[] '[IOEff, Terminal] ()

    The type adds the two effects. The same type that the if-then-else expression would have if it would type-check.

    Termination is wiped out since <|> has been redefined under the "RebindableSyntax" extension to cancel any Termination effect of the first term


    Ooops... No. That latter type is not good, since I can cheat the compiler telling that I paid when I don't:

    (do guard False; pay) <|> sendStuff

    This is the same old problem that I had with "empty" and alternatives, now appears again with "guard".

    After some type-level list processing, the type of test11 is the desired:
    *Main> :t   test11
      :: (Eq a, Num a) => a -> T '[] '[Maybe Terminal, Maybe IOEff] ()

    I told that one of the requirements is to allow the definition of contraints between effects defined by the programmer like the pay-> sendStuff ordering.But these are a zero level kind of requirement.

    Even though I thought it couldn't be possible, it's possible to give the programmer detailed control of invariants of the domain problem using self-defined effects by making use of the rebindable syntax, so that the programmer can redefine the monadic,applicative etc operators using the already defined operators as base:

    {-# LANGUAGE RebindableSyntax, TypeFamilies, TypeOperators,DataKinds,RankNTypes, ScopedTypeVariables #-}
    import  Prelude  hiding ((>>=),(>>),return,concat)
    import qualified Prelude as P 
    import qualified Transient.TypeLevel.Effects as Eff
    import Transient.TypeLevel.Base 
    import Transient.TypeLevel.Move
    import Transient.TypeLevel.Indeterminism
    -- Transient.TypeLevel.Effects has >>= redefined using  rebindable syntax too
    data MyEffect1
    data MyEffect2
    -- my own bind ,  for my problem, with more effects rules defined by me. 
    -- He can use the rules already defined in the Eff module to manage effects to sum, subtract, 
    -- test inclusion etc
    -- In this case, not redefined
    (>>=) = (Eff.>>=)
    I'm using transient in a project with web services and I spent some days dealing with timeouts and reconnections in HTTP/1.1 with TLS. and also an obnoxious bug with the interpretation of the command line parameters. In certain precise circumstances and with some parameters these problems produced so crazy outputs that made me think that I was dealing with some fundamental flaw of the execution model.
    At the end I was just a matter of bad management of two varables and some unnecessary complications in the code.
    I learned one thing or two. The HTTP code is producing more headaches than anything else in transient. I will upload a new version as soon as I can.
    watched the superb talk of alexis king about performance and effects.

    This talk inspires me some considerations about effect systems, continuations and performance:
    I wanted continuations to be given by the runtime instead of in programmer space in a continuation monad. In fact I worked in ETA to redefine it so that the IO monad delivers it. That was my dream since many effects are not possible without continuations. But having continuations as a monad in a monadic stack add extra complexity and extra performance loss.

    I tried to define transient so that the primitives are independent of how the effects are implemented, as long as they provide state and continuations over Applicative/Alternative/Monad instances. In fact I have two different engines. The second one uses a different continuations monad and exceptions to implement the Alternative instance. When the patch of Alexis become available, the translation of transient to wathever monad that uses it is relatively straigforward.

    In the meantime, The current monad does not add a lot of boilerplate. The monad implements continuations maintaining a stack of bind parameters which is lazily constructed and discarded when not used. It is possible to use noTrans, to use the underlying monad, when I write ordinary code without async, empty, react or distributed computing. The bind in this case is as fast as one of the ordinary state monad. What may be slower is getState and setState. It is the price to pay for the flexibility of intruducing and retrieving any variable state at any moment, as many of them as you please, without recompiling/redefining/adding instances runners and/or handlers anywhere in the rest of the modules that may be involved. However that performance of state is similar to the best of the extensible record libraries i guess.
    To handle many states is the main chore of monadic stacks and effect systems. since the rest of the effects have limited number of them, but states can be unlimited. However in Transient the need of state is drastically reduced, since basically data is in scope in the form of ordinary variables:
    let  x=  MyData.....
    y <-  multireaded,distributed,streaming...
    --   x still in scope! (even if now it is in a different machine!!)
    Concerning effects, there are different views of what an effect is:
    • In effect libraries, an effect is associated with an interpreter/handler which is called when the effect is needed
    • In Final Tagless/MTL libraries, they are monad transformers in a stack.
    • In libraries like RIO they are class instances that group methods.
    • In Transient, in the type-level transient that I'm developing, effects are salient features of the computation

    The first three can aggregate effects in the type signature and this is good for documentation purposes and to tell the type system what and how the runners and lifters should be inserted in the code to make it type match.

    I think the main functionality of type-level lists of effects in effect libraries is to make type signatures less ugly than monad transformers and to make sure that the runner of each effect is present. and Oleg Kiselyov designed them originally for that purpose. In the case of MTL, the monadic stack signatures are a consequence of his very structure, and in the case of Has.... strategy, it is intended for documentation purposes, to make the program more readable.

    In Transient since the effect can be created with a simple data declaration without runtime representation, they can be used to check the compliance of domain-specific requirements at compile time using type-level logic, as the examples written here above detail.

    In transient any other effect can be created by the combination of the already existent. I don't provide a reader of writer effect since getRData/setRData is equivalent to the reader/writer effects.
    I have to create a table of how-to-do features based on
    I defined whileException:
    -- re execute the first argument as long as the exception is produced within the argument. 
    -- The second argument is executed before every re-execution
    -- if the second argument executes `empty` the execution is aborted.
    whileException :: Exception e =>  TransIO b -> (e -> TransIO())  -> TransIO b 
    whileException mx fixexc =  mx `catcht` \e -> do fixexc e; whileException mx fixexc
    the IO version would be exactly the same, but using catch instead of catchc.

    The motivation is that sometimes it is necessary to retry some task after some fix that tries to solve the problem:

         `whileException` \theException-> fixTheException

    I had this problem with sockets. when a connection is closed by the remote node, the program is not aware of in until he tries to send again, so I can not preemtively reopen the channel. so:

    sendSecure  connection toSend= do
          send connection toSend
      `whileException` \(SomeException _) -> reopen connection
    Transient programs seems to me like an assembly line where there are many workers doing certain precise task, everyone is busy at the same time, while a program with a traditional execution model seems to me like an assembly line with a single man which build the car from the beginning to the end. He may give work to other workers while he is waiting, but, since he is coordinating, many tasks that may be done in parallel can not be done and only a single car can be assembled in the line each time.
    The next car is waiting for the man to finish the previous car and run to the beginning of the assembly line in a loop. He can not ask for something
    [He can not ask for something] <- unfinished phrase
    This message was deleted
    Alberto @agocorona 01:12
    Dialog between a Web server and myself:
    • Me: Why you don't answer my second request after 20 seconds? We agreed that the connection keeps open: I said HTTP/1.1 and you told me "Connection: Keep-Alive"...The connection is open,I know, you receive my request, but you don't answer me after 20 seconds. Why you don't close the connection to signal that you don't want anything more trough this channel?
    • Serv: If you make the request through a Web Browser you will know.
    • Me: Okay, making the request with chrome, you answer that your keep-alive connections have a timeout of 15 seconds. Why don't you say It to the Transient application?
    • Serv: You don't send headers. I like headers...
    • Me: Okay, so I have to send to you "Connection: Keep-Alive"? HTTP/1.1 in the first line says so implicitly...By the way, why don't you simply close the connection or, else send me the timeout info? Why aren't you more explicit in the response?
    • Serv: I am a server and I like headers
    • Me: Okay take it: "Connection: Keep-Alive"
    • Serv: OK "Keep-Alive: 15, max=500"
    • Me: Ok, now my program know that you keep the connection open indefinitely, but after 15 seconds or 500 requests, you won't answer
    • Serv: Yes. I like headers
    Hugh JF Chen
    @agocorona That's why I use req as the http client within my transient service and ask you if we can embed a http server, like warp to expose the transient service.
    @hughjfchen You're right. I have been thinking about leaving this fight. I recommended you at the beginning to use other packages for the HTTP stuff. Integrating another webserver directly in transient is not possible since the communication node-node uses another protocol. The transient server should manage both protocols because the integration with web services is a practical need at the end of the day and is a priority. On the other side, that allows me to use transient primitives for parsing, process management, and also enjoy solving problems in interesting ways.
    Do you use communications node-to-node in your case? runAt etc?
    @hughjfchen Do you use communications node-to-node like runAt etc?

    About backtracking and exceptions
    Going through the code I found this commentary (3 years ago!) from @harendra-kumar that I haven´t noticed

    -- XXX Should we enforce retry of the same track which is being undone? If the
    -- user specifies a different track would it make sense?

    Transient uses a kind of multi-level backtracking to solve a variety of problems, like closing open resources, undoing transactions, or even navigating recursive structures. Also, it is used for the management of exceptions. This backtracking is different from the sigle level backtracking of the alternative.

    As you can see in the example of the tutorial. ( I translated the undo track to am exception Undo. That is semantically similar, with the advantage that this allows to backtrack haskell exceptions when they happens anywhere in the code.

    #!/usr/bin/env stack
    -- stack --install-ghc --resolver lts-6.23 runghc   --package transient 
    import Control.Monad.IO.Class (liftIO)
    import Transient.Base
    import Transient.Backtrack
    instance Exception Undo where
    main= keep $ do
           liftIO $ print "done!"
    productNavigation = liftIO $ putStrLn "product navigation"
    reserve= liftIO (putStrLn "product reserved,added to cart")
                     `onException'` \Undo -> liftIO (putStrLn "product un-reserved")
    updateDB=  liftIO  (putStrLn "update other database necesary for the reservation")
                     `onException'` \Undo -> liftIO (putStrLn "database update undone")
    payment = do
               liftIO $ putStrLn "Payment failed"
               throw  Undo

    Here undo initiates the process of backtracking due to the failure in the payment. This executes from bottom to top the second argument of the onException' until a restarting of the normal flow is found (or the thread stop, as is in this case)

    It makes sense that each onException' does not manipulate this backtracking BY DEFAULT since this enhances composability and modularity. Imagine the real case, with real stuff, databases etc; These steps may be in different modules, in different libraries, maybe by different teams or even companies. Does it make sense to pass to them the responsibility for how the backtracking should continue? It is better to leave this decision to the application programmer which initiates the undo.

    Ok but how the application programmer can taylor the backtracking to his needs?

    onException $ \Undo -> do
                 resp <-         option "retry" "retry all that was undone" <|> 
                                 option "stop" "no further actions" <|>
                                 option "back" "continue backtracking" <|>
                                 option "other"  "throw a different exception" 
                                 option  "bother" "backtrack with other track"
                 case resp of
                          "retry" -> continue
                          "stop"  -> empty
                          "back" -> backtrack
                          "other"-> throw SomeOtherException
                          "bother" ->  back Other

    More succinctly (at the cost of some readability):

                                 (do option "retry" "retry all the undone"; retry) <|> 
                                 (do option "stop" "no further actions"     ; empty) <|>
                                 (do option "back" "continue backtracking" ; backtrack) <|>
                                 (do option "other"  "throw a different exception" ; throw SomeOtherException) <|> 
                                 (do option  "bother" "backtrack with other track"  ; back Other)
    These are the options. The three first are there since the beginning, and the default is backtrack. The other two were close to work reliably, but in the case of exceptions there was some infinite loop when the exception re-raised was of the same type than the original exception. After adding some more code and blowing my mind with many balls in the air: exceptions, continutions, threads... I found that eliminating some code and moving some lines the problem was solved, now working in my box and I will upload it as son as I finish some tests. adn fix some other bugs.
    If you don't want to backtrack and want to use the traditional semantics for exceptions catcht works like catch in the transient monad.
    this program prints "world2..callstack..."
    main= keep' $ do
        onException $ \(SomeException e) -> do liftIO (print e); empty
        onException $ \(SomeException e) -> error "world2"
        onException $ \(SomeException e) ->error "world"
        error "hello"
    onException... is the unary equivalent of onException'...
    here each onException generates a new exception that is backtracked.
    when the alexis patch become available It will be straightforward to use it in Transient to make the transient monad as fast as Haskell can be.
    continue return Unit (()). Sometimes it is necessary to return values of other type to resume execution. In this example a Int is raised and the handler will return the next number:
    import Transient.Base
    import Control.Exception hiding (onException)
    instance Exception Int
    main= runTransient $ do 
         x<- return 1 `onException'` \n -> continue >> return (n+1)
         liftIO $ print (x:: Int)
         throw x
         return () 
    > program
    if continue is omitted, the program would backtrack after the handler. Since there is no further handlers it will print "1" and that's all.
    runTransientis fine for running non-interactive single threaded programs.
    No new fancy facts found to tell. Fighting fails furiously.
    Released new versions of transient, transient-universe to hackage after a long time:
    looking forward to integrate Axiom in codeworld