Transient, Transient Universe, HPlay/Lobby

Everything about these libraries that makes everithing infinitely composable https://github.com/transient-haskell/transient/blob/master/README.md

Hugh JF Chen

@hughjfchen

Hi, @agocorona , line between 1390 and 1391 are a comment block, not sure where to insert these tow lines code:

1381      _ -> do
1382            let uri'= BC.tail $ uriPath uri !> uriPath uri
1383            if  "api" `BC.isPrefixOf` uri'
1384              then do
1385                {-
1386                 check for len
1387                 if len entonces normal
1388                    check for url encoded
1389                    check for json
1390 
1391                 else check for chunked
1392                 lookup "Transfer-Encoding" headers of
1393                          Just "chunked" ->
1394 
1395                -}
1396 
1397                log <- return $ Exec:(Var $ IDyns $ BS.unpack method):(map (Var . IDyns ) $ split $ BC.unpack $ BC.drop 4 uri')
1398

1374      "CLOS" ->
1375           do
1376            conn <- getSData
1377            sendRaw conn "OK"                                   !> "CLOS detected"
1378 
1379            mread conn
1380 
1381      _ -> do
1382            let uri'= BC.tail $ uriPath uri !> uriPath uri
1383            if  "api" `BC.isPrefixOf` uri'
1384              then do
1385                {-
1386                 check for len
1387                 if len entonces normal
1388                    check for url encoded
1389                    check for json
1390 
1391                 else check for chunked
1392                 lookup "Transfer-Encoding" headers of
1393                          Just "chunked" ->
1394 
1395                -}
1396 
1397                log <- return $ Exec:(Var $ IDyns $ BS.unpack method):(map (Var . IDyns ) $ split $ BC.unpack $ BC.drop 4 uri')
1398 
1399 
1400 
1401                headers <- getHeaders
1402                maybeSetHost headers
1403                return () !> ("HEADERS", headers)
1404                str <-  giveData  <|> error "no api data"
1405                if  lookup "Transfer-Encoding" headers == Just "chunked" then error $ "chunked not supported" else do
1406 
1407                    len <- (read <$> BC.unpack

Hugh JF Chen

@hughjfchen

I went through the source code, looks like that's a request header receiving from a client request. Does that means it is the client who sends a request with a Connection: close header parameter indicating that's the last request through this connection and transient close the connection actively?

Alberto

@agocorona

My bad! . Sorry @hughjfchen the change should be after the line 1401 !!!

1401                headers <- getHeaders
1402                setState $ HTTPHeaders headers             <--------- that insertion
1403                maybeSetHost headers

After this change (and the previous in api) then check if transient closes actively the connections when the request header has Connection: close

Hugh JF Chen

@hughjfchen

hi, @agocorona , there still lots of FIN_WAIT_2 connections

chenjinfendeMacBook-Pro:~ chenjinfen$ netstat -na|grep 9981
tcp4       0      0  192.168.0.106.58601    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58600    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58599    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58598    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58597    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58596    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58595    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58594    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58593    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58592    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58591    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58590    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58589    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58585    120.76.121.79.9981     FIN_WAIT_2 
tcp4       0      0  192.168.0.106.58582    120.76.121.79.9981     FIN_WAIT_2

and at the transient program side, I got these:

keep: Communication error
CallStack (from HasCallStack):
  error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals

I've checked the source code and found the line 1361 as following:

1344   --            let Connection{remoteNode=rnode,localClosures=localClosures,closChildren= rmap} = conn
1345   --            -- TODO How to close Connection by discriminating exceptions
1346   --            mnode <- liftIO $ readIORef rnode
1347   --            case mnode of
1348   --              Nothing -> return ()
1349   --              Just node  -> do
1350   --                            liftIO $ putStr "removing1 node: " >> print node
1351   --                            nodes <- getNodes
1352   --                            setNodes $ nodes \\ [node]
1353   --            liftIO $ do
1354   --                 modifyMVar_ localClosures $ const $ return M.empty
1355   --                 writeIORef rmap M.empty
1356   --            -- topState >>= showThreads
1357 
1358   --            killBranch
1359 
1360 
1361    setData $ (ParseContext (NS.close sock >> error "Communication error" ) input
1362              ::ParseContext BS.ByteString)
1363 
1364    setState conn{connData=Just (Node2Node (PortNumber port) sock addr)}
1365    maybeTLSServerHandshake sock input
1366 
1367 
1368 
1369   --  (method,uri, headers) <- receiveHTTPHead
1370    (method, uri, vers) <- getFirstLine
1371    return () !> (method, uri, vers)
1372    case method of
1373 
1374      "CLOS" ->
1375           do
1376            conn <- getSData
1377            sendRaw conn "OK"                                   !> "CLOS detected"
1378

Hugh JF Chen

@hughjfchen

Looks like transient closed the connection at the server side but the browser doesn't close it at the browser side

this is more clear when I set the transient response to http 1.1 and connection: keep-alive

the browser will keep 2 connection open

and send following requests through these connections but at the transient side these two connections had been closed so transient got a communication error and the browser never got a response

It just says waiting for server

Hugh JF Chen

@hughjfchen

I think I found something

looks like transient API can't handle multi connections

If there're multi connections to my transient api program,the first time browser can get result, however, following requests the browser just keep waitting for server and can't get response. meanwhile, the transient api program will print out communication error:

keep: Communication error
CallStack (from HasCallStack):
  error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals

looks like transient doesn't know which connection it should use to get back to the browser?

So if connection gets closed immediately after request, it can handle requests correctly because there is only one connection at any time

Hugh JF Chen

@hughjfchen

However, looks like browser loves to keep multi connections open to the server

they don't play well

Heath Matlock

@heath

the first 4 links to articles in the documentation section for transient-universe are broken

Hugh JF Chen

@hughjfchen

looks like http clients(curl, browsers) won't send the Connection: close request header

Hugh JF Chen

@hughjfchen

I checked the HTTP spec, for http 1.0, looks like we need to close the connection after response, no need to check the request header for connecton: close

Hugh JF Chen

@hughjfchen

I changed the api to following:

myApi :: TransIO BS.ByteString -> Cloud ()
myApi w= Cloud $ do

    Log rec _ _ _ <- getState <|> return (Log False [][] 0)
    if not rec then empty else do
       HTTPHeaders hdrs <- getState <|> error "api: no HTTP headers???"
       let closeit= lookup "Connection" hdrs == Just "close"
       conn <- getState  <|> error "api: Need a connection opened with initNode, listen, simpleWebApp"
       let send= sendRaw conn

       r <- w
       return () !> ("response",r)
       send r                         --  !> r
       -- when closeit $ liftIO $ mclose conn
       liftIO $ mclose conn

and now connections are in TIME_WAIT state, which is ok:

chenjinfendeMacBook-Pro:~ chenjinfen$ netstat -na|grep 9981
tcp4       0      0  192.168.0.106.60035    120.76.121.79.9981     ESTABLISHED
tcp4       0      0  192.168.0.106.60006    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60010    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60011    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60012    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60013    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60014    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60015    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60016    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60017    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60020    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60022    120.76.121.79.9981     TIME_WAIT  
tcp4       0      0  192.168.0.106.60032    120.76.121.79.9981     TIME_WAIT

those connection in TIME_WAIT state won't use resource and will close very soon

But that's only a quick and dirty hack

Better way should take the http version into account

if http 1.0, close it immediately after response; if http 1.1, keep-alive and after some requests or timeout the connections will get closed.

Hugh JF Chen

@hughjfchen

Even I close the connection immediately after response, I still get the following error at the transient program:

keep: Communication error
CallStack (from HasCallStack):
  error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals

Not sure why

Hugh JF Chen

@hughjfchen

Did some more tests. Transient as client may reuse the connection, but transient as server doesn't reuse connections.

In the api case

If client and server both are transient, it does reuse connection(I found there is always only one connection between them). However, it won't reconnect if the connection drops

Hugh JF Chen

@hughjfchen

I guess some effect has to be put to improve the connection management for transient.
1.as Client
2.as Server
3 both client and server

Alberto

@agocorona

thanks @hughjfchen I will look detaily at it

Alberto

@agocorona

reconnections will be available in the next release. I need a timeout to close connections too.

Hugh JF Chen

@hughjfchen

reconnections will be available in the next release. I need a timeout to close connections too.

Great to hear that. Thank you very much for your effort to develop and maintain transient.

Alberto

@agocorona

@hughjfchen thanks to you for your feedback!

I see that at least my version of curl uses http 1.1 and does not request close connection on the header

Alberto

@agocorona

And, as you say, the connection stay even if called from a remote node

Alberto

@agocorona

Even I close the connection immediately after response, I still get the following error at the transient program:
keep: Communication error
CallStack (from HasCallStack):
  error, called at src/Transient/Move/Internals.hs:1361:46 in transient-universe-0.5.2-KE58pdlDpxjCHxvpKKKt4i:Transient.Move.Internals

Yes I see. the connection logic assumes a permanent connection between transient nodes (like Erlang nodes etc) This logic can not be assumed for REST services. I will take some time to fix the problem of reconnections (almost done) and timeouts which need some cleanup of closures, not in the case of REST , but in the case of using distributed primitives like runAt etc.

Alberto

@agocorona

For the time being, I have a version that closes the connection either if the HTTP header has version 1.0 . If the version is 1.1 and has header "Connection: close" it closes the connection

I will upload it today. Could you get along with this for the moment?

Alberto

@agocorona

uploaded https://github.com/transient-haskell/transient-universe

Hugh JF Chen

@hughjfchen

Hi, @agocorona Thanks for your prompt update. Sure,I'll try and give feedback as soon as possible

Hugh JF Chen

@hughjfchen

Hi, @agocorona , Is this update version still needs put sync before runCloud?

Alberto

@agocorona

yes it does.

Alberto

@agocorona

perhaps instead of sync in every asynchronous call it would be better to drop it and use:

api $ threads 0 $  gets <|> badRequest...

Since everithing would be synchronous, since all would be executed in the current thread. But I haven`t tested it

Where communities thrive

People

Repo info

Activity