by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Ohad Shai
    @oshai
    btw - what is the configured connection timeout?
    Arunachalam Sockalingam
    @asockalingam
    2 sec
    but it adds a 30 sec buffer by default I guess
    from the code it looks like it.
    Ohad Shai
    @oshai
    for this message? was not cleaned by connection as it should, will destroy it
    Arunachalam Sockalingam
    @asockalingam
    yeah
    holder.timeElapsed > configuration.queryTimeout + extraTimeForTimeoutCompletion
    extraTimeForTimeoutCompletion is 30 sec by default
    Ohad Shai
    @oshai
    yes, this mechanism is a backup for the timeout and it should not operate unless there is a bug
    Arunachalam Sockalingam
    @asockalingam
    probably reduce it to 1 second and try ?
    Ohad Shai
    @oshai
    it will kill connections, but since it can't establish new connections that will not help
    Arunachalam Sockalingam
    @asockalingam
    for one db we have pg bouncer and if this happens it able to recover
    but this one we dont and it is not able to recover
    how to destroy and same establish connection?
    probably evict the connection right
    Ohad Shai
    @oshai
    not sure I understand the question, once connection is killed a new one is created where you try to run a query
    that is the pool responsibility
    Arunachalam Sockalingam
    @asockalingam
    ok once destroyed it should create a new connection right?
    why would that not happen?
    Ohad Shai
    @oshai
    yes
    it happens,
    that is why you see numbers rolling up
    the issue is the when it creates new connection
    Arunachalam Sockalingam
    @asockalingam
    but it is this vicious cycle
    Ohad Shai
    @oshai
    it tries to test if it can query the db with test query and it fails
    Arunachalam Sockalingam
    @asockalingam
    also there is no exception just logs.
    Ohad Shai
    @oshai
    that is why traces might help
    Arunachalam Sockalingam
    @asockalingam
    cool. let me try with trace.
    Ohad Shai
    @oshai
    we will be able to see netty events that were recieved
    Arunachalam Sockalingam
    @asockalingam

    @oshai

    I think time out is not respected

    see this again happened on prod

     logger_name: com.github.jasync.sql.db.pool.ActorBasedObjectPool
       message: received message: Take @35969632 ; availableItems=0 waitingQueue=122 inUseItems=10 inCreateItems=0 ActorCoroutine{Active}@61c3996
    inUseItems -- all 10 queries should have been kicked out right?
    just happened in one node.
    Arunachalam Sockalingam
    @asockalingam
    we have a 1second time out at database level and I don't see even that being respected.
    Arunachalam Sockalingam
    @asockalingam
    it is not able to make the connection or create connection can we fail fast ? and try to re-connect ?
    Ohad Shai
    @oshai
    can you share the full log with me? I think the timeout in the db is unrelated
    Arunachalam Sockalingam
    @asockalingam
    I only enabled logs for com.github.jasync.sql.db.pool and I have sent u the logs.
    Arunachalam Sockalingam
    @asockalingam

    My suspect this code

    ```

    something is wrong with establishing new connections is my doubt.
    Arunachalam Sockalingam
    @asockalingam
    I was able to connect manually from the Node via psql.
    also, none of test to validate is throwing an exception instead they are failing silently.
    Ohad Shai
    @oshai
    about the line you pointed out there is a condition totalItems < configuration.maxObjects that should prevent from just creating more and more connections
    about the tests, if you can point out to a specific problem it would be helpful
    Arunachalam Sockalingam
    @asockalingam
    Arunachalam Sockalingam
    @asockalingam
    I think this is the problem
    Ohad Shai
    @oshai
    Interesting...
    Ohad Shai
    @oshai
    I am not sure it is related, I think they did a fix in their own internal wrappers of netty, not netty itself
    Arunachalam Sockalingam
    @asockalingam
    yeah but the problem is similar.
    Arunachalam Sockalingam
    @asockalingam