AnyCable/Lobby

Vladimir Dementyev

@palkan

@bignoggins :+1:

@proudlygeek Switching to 1.0.1 fixes this problem

(it's gonna be default for Heroku in a few minutes, I'm currently releasing it)

Gianluca Bargelli

@proudlygeek

You rock @palkan :fire: :fire: :fire:

Vladimir Dementyev

@palkan

@PrathapChronus Subscribing to the same stream with different channel ids has been also fixed in 1.0.1

Gianluca Bargelli

@proudlygeek

@palkan, it seems to bind correctly with 1.0.1 but now I'm getting this error instead:

2020-07-07T16:23:26.689770+00:00 app[web.1]: E 2020-07-07T16:23:26.689Z context=ws Websocket session initialization failed: Application error:
2020-07-07T16:23:26.692445+00:00 app[web.1]: E, [2020-07-07T16:23:26.692370 #12] ERROR -- : [AnyCable sid=ec9aed85-622a-425b-af1e-e368eee36b7c] Request origin not allowed: https://dash.callbell.eu

Weird because I have the same exact application in staging with the same formation and it works

Vladimir Dementyev

@palkan

What about action_cable.allowed_origins configuration? This exception comes from Action Cable's built-in forgery protection mechanism (which didn't really work in AnyCable 0.6.x).

Gianluca Bargelli

@proudlygeek

I reverted to what I had for 0.6.x:

config.action_cable.allowed_request_origins = ['https://dash.callbell.eu', 'https://ws.callbell.eu']
config.action_cable.url = "wss://#{ENV['CABLE_URL']}"

But maybe it works now because it doesn't :joy:

But yeah, it's live and running! :tada:

Vladimir Dementyev

@palkan

:tada:

Jerry Shen

@bignoggins

@palkan im running through the heroku deploy docs, is this a typo?

# Don't forget to add RAILS_ENV if using Rails
heroku config:set RAILS_ENV=production -a anycable-demo-rpc

there is no anycable-demo-rpc project mentioned anywhere else in the docs. Should that be

# Don't forget to add RAILS_ENV if using Rails
heroku config:set RAILS_ENV=production -a example-app-anycable

Jerry Shen

@bignoggins

Getting this error when I deploy to Heroku

2020-07-12T06:28:32.172559+00:00 app[web.1]: E, [2020-07-12T06:28:32.172467 #7] ERROR -- : [AnyCable sid=79b519cf-7a67-4f9c-8d8a-20e79d042b8a] undefined method `call' for ApplicationCable::Connection:Class

any ideas @palkan

Jerry Shen

@bignoggins

^ turns out it was because my RAILS_ENV was set to staging. I guess anycable requires it to be development or production?

Vladimir Dementyev

@palkan

It could be any RAIL_ENV; make sure you have adapter: any_cable in your cable.yml for the specified env

there is no anycable-demo-rpc project mentioned anywhere else in the docs

Yeah, looks like a typo. Will fix it. Thanks!

Jerry Shen

@bignoggins

@palkan gotcha yeah added staging in the cable.yml and that fixed it as well thanks

Javier Borrego Cejudo

@Ronjea

Hi There is anyone trying to connect to anycable using the https://www.npmjs.com/package/action-cable-react-jwt I´m having problems due to the treshold inside of the library I increased that to be greater than 6 seconds.

Is there any configuration in anycable that affects or does not allow who connects to reconnect?

I´m having the connection and the backend is broadcasting but is not sending any disconnection messages.

So the library determines the connection is staled due we do not receive any pings. I tried with WS in javascript directly the problem is the same the connection is hold and the backend is broadcasting but I´m not receiving any data from it

Vladimir Dementyev

@palkan

If you’re not receiving ping messages then the client is likely not connected.

Could you please check anycable-go logs (ideally, with ANYCABLE_DEBUG=1).

Javier Borrego Cejudo

@Ronjea

@palkan Where do you add this anycable_debug ?

Vladimir Dementyev

@palkan

You can add as env var or run anycable-go -debug

Javier Borrego Cejudo

@Ronjea

will do it tomorrow I will post here some stuff

Thanks a lot @palkan !

Javier Borrego Cejudo

@Ronjea

@palkan I think we figure it out something.

When we do reach 100% in CPU consumption it closes the connection and will never respond to any messages.
This is the log and the information I got from it
This is the header

I 2020-07-20T04:10:31.011Z context=main Starting AnyCable 1.0.0.preview1 (pid: 14513, open file limit: 1024)
D 2020-07-20T04:10:31.011Z context=disconnector Calls rate: 10ms
I 2020-07-20T04:10:31.012Z context=main Handle WebSocket connections at /cable
I 2020-07-20T04:10:31.012Z context=main Handle health connections at /health
I 2020-07-20T04:10:31.013Z context=rpc RPC controller initialized: 0.0.0.0:50051 (concurrency: 28, proto_versions: v0,v1)
I 2020-07-20T04:10:31.013Z context=http Starting HTTPS server at 0.0.0.0:3334
I 2020-07-20T04:10:31.013Z context=pubsub Subscribed to Redis channel: __anycable_

D 2020-07-20T04:10:31.615Z context=hub stream=my_channel No sessions -- This error message could the reason for when  some channel confirmed and other channel are waiting for the confirmation. couldn't establish any sessions for the same

context=node sid=TYAIr4jzMhG02BT94 Websocket closed: websocket: close 1005 (no status) -- This is error for when user cancelled the connection or when we force fully closed the browser we will loose/close that connection.

Vladimir Dementyev

@palkan

Do you have any logs with context=rpc and context=hub with the same sid value?

These are not errors, just debug messages.

The main question is whether authentication and channel subscription succeed.

Javier Borrego Cejudo

@Ronjea

Checking that right now

Javier Borrego Cejudo

@Ronjea

https://pastebin.com/hXUbZTqr

Here you can see at the moment the websocket dies

Vladimir Dementyev

@palkan

Thanks!
Doesn’t look like pings problem, ‘cause client disconnected 4.5min after the connection.

Did client disconnect on its own or that was a user initiated close?

Vladimir Dementyev

@palkan

And I don’t see any broadcast logs here

Javier Borrego Cejudo

@Ronjea

I disconnected myself because I stopped receiving anything from it

I saw 1005 errors

and sometimes 1006 errors

Benjamin Simmons

@bdsimmons

Does anyone know if anycable supports: https://graphql-ruby.org/subscriptions/broadcast

Prathap B

@PrathapChronus

Hi @palkan,
I was trying to do a load test on a chat server using Artillery.

In server, whenever chat text is received via channel, the chat is stored and then 3 broadcasts happen to 3 different channels. Hence each message triggers 3 broadcasts.

We followed the OS tuning mentioned in https://docs.anycable.io/#/v1/anycable-go/os_tuning and in https://kemalcr.com/blog/2016/11/13/benchmarking-and-scaling-websockets-handling-60000-concurrent-connections/ (referenced in the annuversary post of Anycable).

I was testing with 10000 concurrent connections. No issues with memory/CPU. When the number of WS messages exchanged per second crossed 350 for a period, we experienced significant delay (8-12s) in receiving the notification for the chat created. Assume there is no big delay in the ActiveRecord save. Also during that time, new channel subscriptions took more than 5s.

We tried wih grpc concurrency of 1000 and Go concurrency of 990. Can we increase the number? Or is there some limit which we must not cross?

Vladimir Dementyev

@palkan

grpc concurrency is limited by the size of your DB pool, if you need a db connection for each call (which is likely the case). That could be the reason of delays; try increasing it as well.

@bdsimmons Yes, it does; we have a specific gem for that: https://github.com/anycable/graphql-anycable

@PrathapChronus

I was trying to do a load test on a chat server using Artillery.

Never used it, looks cool. Would be great if you share an example benchmark script.

@PrathapChronus Did you collect instrumentation data? I guess, there should be a spike in rpc_retries, which could indicate the congestion at the RPC level.

Benjamin Simmons

@bdsimmons

@palkan According to the maintainer of graphql-ruby: "yeah, broadcast: true requires updates to the implementation, so if they haven’t done it yet, it won’t do anything yet"

and that feature is only 1 month old in graphql-ruby

so, has it been implemented in anycable?

Andrey Novikov

@Envek

@bdsimmons, no, support for graphql-ruby broadcasts hasn't been implemented yet in graphql-anycable. I will try to get to it next week.

We were thinking about broadcasting when we just started to implement graphql-ruby subscriptions adapter for anycable, but it was quite complicated with a lot of questions to answer (how to understand when we can broadcast and when we can't). Now, when graphql-ruby seems to solve all these concerns, we will be very happy to add support for its broadcasting.

Benjamin Simmons

@bdsimmons

@Envek thanks for the response. I will keep a lookout for it!!

Andrew Le

@andrewcmyk

Hi all, I'm looking for general advice on monitoring/health checks. Our Rails app lost connection to Redis, but the /health endpoint was "stayin alive", so it took a while to notice the outage. Not sure if ensuring the connection to Redis belongs in the RPC health check or elsewhere. To make matters worse, this is running on Heroku and limited to one web process.

I'm thinking maybe I could create an unauthenticated channel and ping that

Andrew Le

@andrewcmyk

I may have spoke too soon... when I repro an unreachable Redis URL, it does actually crash the RPC server

Where communities thrive

People

Repo info

Activity