Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    anbublacky
    @anbublacky
    I have the working version of artillery script, Which can make the server Go up to 200%. Now i am not sure why the problem is occuring with anycable-go
    anycable go version 1.0.2
    anycable version 1.0.1
    Vladimir Dementyev
    @palkan

    Hey @anbublacky!

    Do you know, how many connections are open when CPU hits 200%? How many cores do you have, btw?

    Also, what is the open files limit (you can find in logs “Starting AnyCable-Go ...”).

    Do you collect metrics during the benchmark? Could you share the artillery script, please, as well?

    anbublacky
    @anbublacky

    @palkan
    In production, usually we have 2 core, when we know we will get more people, we increase it to 8 core.
    The testing with artillery happened on staging, with 2 core, The script runs for 1 min and 20 sec, Initial 30 sec CPU is around 50%, after that it keeps increasing and by the time it reaches 1 min 10 sec it attains 200% and returns error.

    Open files limit log
    Starting AnyCable 1.0.2 (pid: 3571, open file limit: 64000)

    Artillery scripts
    websocket_test.yml

    config:
      processor: "./my-functions.js"
      target: "wss://URL/cable?access_token=TOKEN"
      ensure:
        maxErrorRate: 1
      phases:
        - duration: 20
          arrivalRate: 1
          rampTo: 5
          name: "Warming up"
        - duration: 60
          arrivalRate: 10
          rampTo: 100
          name: "Max load test"
    scenarios:
      - engine: "ws"
        flow:
          - function: "createTimestampedObject"
          # - send: "{{ data }}"
          - send: '{"command":"subscribe","identifier":"{\"channel\":\"ConversationChannel\",\"conversationId\":181}"}'
          - think: 3
          - send: '{"command":"message","identifier":"{\"channel\":\"ConversationChannel\",\"conversationId\":181}","data":"{\"conversation_id\":181,\"body\":\"{{data}}\",\"action\":\"create_message\"}"}'

    my-functions.js

    module.exports = { createTimestampedObject };
    
    function createTimestampedObject(userContext, events, done) {
      const subscribe = { "command": "subscribe", "identifier": "{\"channel\": \"ConversationChannel\", \"conversationId\": 181 }"}
      const data = Date.now();
    
      // set the "data" variable for the virtual user to use in the subsequent action
      userContext.vars.data = data;
      userContext.vars.subscribe = subscribe;
      return done();
    }
    Alexander Stupnyk
    @AlexanderStupnyk
    Hello everyone! I can't find in the documentation a way to filter out any sensitive data which may be passed as an argument via AnyCable. Can someone show me how i can do this? Thanks.
    Vladimir Dementyev
    @palkan
    Hi @AlexanderStupnyk! Do you mean log filtering? That could (and should) be done via Rails built-in mechanism (in case you’re using Rails, of course).
    Alexander Stupnyk
    @AlexanderStupnyk
    @palkan Yes, i mean log filtering and i use Rails :). So, as you say, i need to do that in Rails.application.config.filter_parameters. And if my parameters came as json, I need to come up with something more complicated but in the same place?
    Vladimir Dementyev
    @palkan
    Yeah; it should be handled at the logger layer
    Benjamin Klotz
    @tak1n
    Hi @palkan just a short question, are there any plans for anycable (rack version https://github.com/anycable/anycable-rack-server) to try out the new ruby 3 fiber scheduling.
    I just saw https://www.youtube.com/watch?v=Y29SSOS4UOc&feature=emb_title and questioned myself how this performs in comparison to anycable-go or erlycable. Not that I need that (we are actually quite happy with the anycable-go version), but it would just be interesting :)
    Vladimir Dementyev
    @palkan
    In theory (I haven’t tried), you can use async server (falcon) for that, since anycable-rack-server is actually a Rack middleware.
    Probably, using async-websocket for building a WS server would be more efficient.
    Benjamin Klotz
    @tak1n
    Okay, thx I will try that out
    Uzaif
    @uzaif313
    i am facing this issue sometime in production
     DEBUG 2021-01-21T07:39:43.216Z context=rpc Authenticate response: error_msg:"undefined method `call' for ApplicationCable::Connection:Class" 
     ERROR 2021-01-21T07:39:43.216Z context=ws Websocket session initialization failed: Application error: undefined method `call' for ApplicationCable::Connection:Class
    Vladimir Dementyev
    @palkan
    Could you please share you ApplicationCable::Connection class code?
    Uzaif
    @uzaif313
    @palkan this content of my connection code.
    module ApplicationCable
      class Connection < ActionCable::Connection::Base
      end
    end
    Uzaif
    @uzaif313
    @palkan do i need to put something in connection class?
    Nathaniel Woodthorpe
    @d12
    @uzaif313 does this help? anycable/anycable-rails#137
    Nathaniel Woodthorpe
    @d12
    :wave: Anycable has been a breeze to work with!The actioncable interface made it easy to get going, and the docs cover basically every question or snag I hit along the way. Thanks for all the work and support :)
    Vladimir Dementyev
    @palkan
    Thank you 😊
    Andy
    @thatandyrose
    hi all! Quick question. Does AnyCable honour my ActionCable settings in my rails app? For example, if I set config.action_cable.allowed_request_origins and config.action_cable.disable_request_forgery_protection will AnyCable honour those? (I'm not quite getting the architecture obviously!)
    Vladimir Dementyev
    @palkan
    HI! Yes, it will. But only if you configure anycable-go to proxy ORIGIN header (e.g., anycable-go --header=cookie,origin
    Andy
    @thatandyrose
    thanks @palkan ! Can I ask... why is this not on by default? 🤔
    Vladimir Dementyev
    @palkan
    Hm, I would say for historical reasons)
    Miguel Michelson Martinez
    @michelson
    funny I just came here to ask the same question, thanks !
    Artem Shevchenko
    @Ar2emis
    Hi! I am migrating from ActionCable to Anycable. In development everything is fine (I am using docker images of anycable-go), but when i am deploying to aws I get
    context=ws Websocket session initialization failed: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 172.17.0.6:50051: connect: connection refused"
    I guess that nginx is blocking connection but really not sure. Did anyone stuck with something like that?
    nginx config:
    upstream socket { server localhost:8000; } ... server { ... location /cable { proxy_pass http://socket; ... } }
    Vladimir Dementyev
    @palkan
    Hi! Does your RPC server listen on 172.17.0.6:50051? The error says that RPC server is not accessible at the address
    1 reply
    Vladimir Dementyev
    @palkan
    It says they you don’t have curl in your container, nothing related to AnyCable
    1 reply
    Vladimir Dementyev
    @palkan
    Adrien Montfort
    @such1_gitlab
    Hi there! I'm trying to deploy AnyCable in production and it seems I need more than 1 RPC server to answer all users connection requests. I've been fiddling around in my k8s cluster for hours and I can't make it so that more than one RPC server receives requests from anycable-go. I've played around with the rpc_pool_size param but with no luck. It seems that anycable-go establishes an RPC connection once at startup and cannot load balance between multiple RPC servers. Is that the case? Is there a way around that?
    Vladimir Dementyev
    @palkan
    Hi! Yeah, to use multiple RPC servers with with a single anycable-go server you need to add a load balancer (envoy, nginx, etc.)

    it seems I need more than 1 RPC server to answer all users connection requests

    What do you think so? Have you taken a look at the anycable-go metrics (especially, rpc_pending_num)?

    Adrien Montfort
    @such1_gitlab
    Because I can see the only RPC server that responds is overloaded (100% CPU)
    Vladimir Dementyev
    @palkan
    What about memory usage?
    Adrien Montfort
    @such1_gitlab
    Not sure
    Vladimir Dementyev
    @palkan
    Or the RPC calls rate?
    Adrien Montfort
    @such1_gitlab
    I'd have to simulate as many clients as I have in production
    But I don't have the tooling for that just yet
    I haven't found a way to parallel run ActiveCable and AnyCable
    Vladimir Dementyev
    @palkan

    I haven't found a way to parallel run ActiveCable and AnyCable

    The only way to do that is:

    • run seprates services / clusters for ActionCable servers (standalone mode) and AnyCable-Go
    • build a custom ActionCable adapter that would broadcast to both Action Cable and AnyCable (kinda mirror adapter)
    • use feature-toggling to set the action-cable-url meta tag
    I never tried this though
    maybe, avoiding a standalone Action Cable server could work out 🤔
    Juan Trejo
    @trejo08
    hey guys
    I have an issue trying to setup anycable inside a kubernetes cluster
    Vladimir Dementyev
    @palkan
    Hi Juan! It would be great if you clarify the problem)
    Conan Jen
    @conanjen
    hello, i was wondering if there was any documentation around scaling the anycable-rails side of things (specifically the rpc server that's run via bundle exec anycable ...)? for context, i've been attempting to load test our current setup in heroku (one anycable dyno running bundle exec anycable --server-command="anycable-go", and one web server just running the normal rails server via puma) and once we hit about 2k connections we start to run into a situation where the actioncable server side of things cannot serve the /cable endpoint quickly enough (the initial requests to /cable don't receive a response). is there something i'm missing since it seems like this should not be remotely anywhere close to load that would cause the rpc server to fail
    16 replies
    Vladimir Dementyev
    @palkan
    Increasing concurrency (at both ends, RPC and Go) is the only way (not considering adding more instances); what about your DB pool size? Did you increase it accordingly?
    The only way to deal with the thundering herd problem is to introduce a random backoff during reconnection (btw, this feature will be in Rails 7 out-of-the-box)
    Conan Jen
    @conanjen
    yeah, we increased the db pool size accordingly as well (i assume the rpc worker pool at most will take one db connection per worker so right now the db pool == rpc pool size, and rpc concurrency is slightly lower than both)
    Conan Jen
    @conanjen
    is there any more settings / logging options that would enable me to see in more detail what might potentially be causing the RPC congestion?
    3 replies