Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Hugh JF Chen
    @hughjfchen
    Hi, @agocorona , line between 1390 and 1391 are a comment block, not sure where to insert these tow lines code:
    1381      _ -> do
    1382            let uri'= BC.tail $ uriPath uri !> uriPath uri
    1383            if  "api" `BC.isPrefixOf` uri'
    1384              then do
    1385                {-
    1386                 check for len
    1387                 if len entonces normal
    1388                    check for url encoded
    1389                    check for json
    1390 
    1391                 else check for chunked
    1392                 lookup "Transfer-Encoding" headers of
    1393                          Just "chunked" ->
    1394 
    1395                -}
    1396 
    1397                log <- return $ Exec:(Var $ IDyns $ BS.unpack method):(map (Var . IDyns ) $ split $ BC.unpack $ BC.drop 4 uri')
    1398
    1374      "CLOS" ->
    1375           do
    1376            conn <- getSData
    1377            sendRaw conn "OK"                                   !> "CLOS detected"
    1378 
    1379            mread conn
    1380 
    1381      _ -> do
    1382            let uri'= BC.tail $ uriPath uri !> uriPath uri
    1383            if  "api" `BC.isPrefixOf` uri'
    1384              then do
    1385                {-
    1386                 check for len
    1387                 if len entonces normal
    1388                    check for url encoded
    1389                    check for json
    1390 
    1391                 else check for chunked
    1392                 lookup "Transfer-Encoding" headers of
    1393                          Just "chunked" ->
    1394 
    1395                -}
    1396 
    1397                log <- return $ Exec:(Var $ IDyns $ BS.unpack method):(map (Var . IDyns ) $ split $ BC.unpack $ BC.drop 4 uri')
    1398 
    1399 
    1400 
    1401                headers <- getHeaders
    1402                maybeSetHost headers
    1403                return () !> ("HEADERS", headers)
    1404                str <-  giveData  <|> error "no api data"
    1405                if  lookup "Transfer-Encoding" headers == Just "chunked" then error $ "chunked not supported" else do
    1406 
    1407                    len <- (read <$> BC.unpack
    Hugh JF Chen
    @hughjfchen
    I went through the source code, looks like that's a request header receiving from a client request. Does that means it is the client who sends a request with a Connection: close header parameter indicating that's the last request through this connection and transient close the connection actively?
    Alberto
    @agocorona
    My bad! . Sorry @hughjfchen the change should be after the line 1401 !!!
    1401                headers <- getHeaders
    1402                setState $ HTTPHeaders headers             <--------- that insertion
    1403                maybeSetHost headers
    After this change (and the previous in api) then check if transient closes actively the connections when the request header has Connection: close
    Hugh JF Chen
    @hughjfchen
    hi, @agocorona , there still lots of FIN_WAIT_2 connections
    chenjinfendeMacBook-Pro:~ chenjinfen$ netstat -na|grep 9981
    tcp4       0      0  192.168.0.106.58601    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58600    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58599    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58598    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58597    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58596    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58595    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58594    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58593    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58592    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58591    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58590    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58589    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58585    120.76.121.79.9981     FIN_WAIT_2 
    tcp4       0      0  192.168.0.106.58582    120.76.121.79.9981     FIN_WAIT_2
    and at the transient program side, I got these:
    keep: Communication error
    CallStack (from HasCallStack):
      error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals
    I've checked the source code and found the line 1361 as following:
    1344   --            let Connection{remoteNode=rnode,localClosures=localClosures,closChildren= rmap} = conn
    1345   --            -- TODO How to close Connection by discriminating exceptions
    1346   --            mnode <- liftIO $ readIORef rnode
    1347   --            case mnode of
    1348   --              Nothing -> return ()
    1349   --              Just node  -> do
    1350   --                            liftIO $ putStr "removing1 node: " >> print node
    1351   --                            nodes <- getNodes
    1352   --                            setNodes $ nodes \\ [node]
    1353   --            liftIO $ do
    1354   --                 modifyMVar_ localClosures $ const $ return M.empty
    1355   --                 writeIORef rmap M.empty
    1356   --            -- topState >>= showThreads
    1357 
    1358   --            killBranch
    1359 
    1360 
    1361    setData $ (ParseContext (NS.close sock >> error "Communication error" ) input
    1362              ::ParseContext BS.ByteString)
    1363 
    1364    setState conn{connData=Just (Node2Node (PortNumber port) sock addr)}
    1365    maybeTLSServerHandshake sock input
    1366 
    1367 
    1368 
    1369   --  (method,uri, headers) <- receiveHTTPHead
    1370    (method, uri, vers) <- getFirstLine
    1371    return () !> (method, uri, vers)
    1372    case method of
    1373 
    1374      "CLOS" ->
    1375           do
    1376            conn <- getSData
    1377            sendRaw conn "OK"                                   !> "CLOS detected"
    1378
    Hugh JF Chen
    @hughjfchen
    Looks like transient closed the connection at the server side but the browser doesn't close it at the browser side
    this is more clear when I set the transient response to http 1.1 and connection: keep-alive
    the browser will keep 2 connection open
    and send following requests through these connections but at the transient side these two connections had been closed so transient got a communication error and the browser never got a response
    It just says waiting for server
    Hugh JF Chen
    @hughjfchen
    I think I found something
    looks like transient API can't handle multi connections
    If there're multi connections to my transient api program,the first time browser can get result, however, following requests the browser just keep waitting for server and can't get response. meanwhile, the transient api program will print out communication error:
    keep: Communication error
    CallStack (from HasCallStack):
      error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals
    looks like transient doesn't know which connection it should use to get back to the browser?
    So if connection gets closed immediately after request, it can handle requests correctly because there is only one connection at any time
    Hugh JF Chen
    @hughjfchen
    However, looks like browser loves to keep multi connections open to the server
    they don't play well
    Heath Matlock
    @heath
    the first 4 links to articles in the documentation section for transient-universe are broken
    Hugh JF Chen
    @hughjfchen
    looks like http clients(curl, browsers) won't send the Connection: close request header
    Hugh JF Chen
    @hughjfchen
    I checked the HTTP spec, for http 1.0, looks like we need to close the connection after response, no need to check the request header for connecton: close
    Hugh JF Chen
    @hughjfchen
    I changed the api to following:
    myApi :: TransIO BS.ByteString -> Cloud ()
    myApi w= Cloud $ do
    
        Log rec _ _ _ <- getState <|> return (Log False [][] 0)
        if not rec then empty else do
           HTTPHeaders hdrs <- getState <|> error "api: no HTTP headers???"
           let closeit= lookup "Connection" hdrs == Just "close"
           conn <- getState  <|> error "api: Need a connection opened with initNode, listen, simpleWebApp"
           let send= sendRaw conn
    
           r <- w
           return () !> ("response",r)
           send r                         --  !> r
           -- when closeit $ liftIO $ mclose conn
           liftIO $ mclose conn
    and now connections are in TIME_WAIT state, which is ok:
    chenjinfendeMacBook-Pro:~ chenjinfen$ netstat -na|grep 9981
    tcp4       0      0  192.168.0.106.60035    120.76.121.79.9981     ESTABLISHED
    tcp4       0      0  192.168.0.106.60006    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60010    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60011    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60012    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60013    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60014    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60015    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60016    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60017    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60020    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60022    120.76.121.79.9981     TIME_WAIT  
    tcp4       0      0  192.168.0.106.60032    120.76.121.79.9981     TIME_WAIT
    those connection in TIME_WAIT state won't use resource and will close very soon
    But that's only a quick and dirty hack
    Better way should take the http version into account
    if http 1.0, close it immediately after response; if http 1.1, keep-alive and after some requests or timeout the connections will get closed.
    Hugh JF Chen
    @hughjfchen
    Even I close the connection immediately after response, I still get the following error at the transient program:
    keep: Communication error
    CallStack (from HasCallStack):
      error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals
    Not sure why
    Hugh JF Chen
    @hughjfchen
    Did some more tests. Transient as client may reuse the connection, but transient as server doesn't reuse connections.
    In the api case
    If client and server both are transient, it does reuse connection(I found there is always only one connection between them). However, it won't reconnect if the connection drops
    Hugh JF Chen
    @hughjfchen
    I guess some effect has to be put to improve the connection management for transient.
    1.as Client
    2.as Server
    3 both client and server
    Alberto
    @agocorona
    thanks @hughjfchen I will look detaily at it
    Alberto
    @agocorona
    reconnections will be available in the next release. I need a timeout to close connections too.
    Hugh JF Chen
    @hughjfchen

    reconnections will be available in the next release. I need a timeout to close connections too.

    Great to hear that. Thank you very much for your effort to develop and maintain transient.

    Alberto
    @agocorona
    @hughjfchen thanks to you for your feedback!
    I see that at least my version of curl uses http 1.1 and does not request close connection on the header
    Alberto
    @agocorona
    And, as you say, the connection stay even if called from a remote node
    Alberto
    @agocorona

    Even I close the connection immediately after response, I still get the following error at the transient program:

    keep: Communication error
    CallStack (from HasCallStack):
      error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals

    Yes I see. the connection logic assumes a permanent connection between transient nodes (like Erlang nodes etc) This logic can not be assumed for REST services. I will take some time to fix the problem of reconnections (almost done) and timeouts which need some cleanup of closures, not in the case of REST , but in the case of using distributed primitives like runAt etc.

    Alberto
    @agocorona
    For the time being, I have a version that closes the connection either if the HTTP header has version 1.0 . If the version is 1.1 and has header "Connection: close" it closes the connection
    I will upload it today. Could you get along with this for the moment?
    Hugh JF Chen
    @hughjfchen
    Hi, @agocorona Thanks for your prompt update. Sure,I'll try and give feedback as soon as possible
    Hugh JF Chen
    @hughjfchen
    Hi, @agocorona , Is this update version still needs put sync before runCloud?
    Alberto
    @agocorona
    yes it does.
    Alberto
    @agocorona

    perhaps instead of sync in every asynchronous call it would be better to drop it and use:

    api $ threads 0 $  gets <|> badRequest...

    Since everithing would be synchronous, since all would be executed in the current thread. But I haven`t tested it