Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 16:12
    jondubois closed #512
  • 16:12
    jondubois commented #512
  • 16:10
    jondubois closed #518
  • 16:10
    jondubois commented #518
  • 16:08
    jondubois closed #530
  • 16:08
    jondubois commented #530
  • 16:05
    jondubois closed #540
  • 16:05
    jondubois commented #540
  • 16:02
    jondubois commented #542
  • 16:01
    jondubois closed #546
  • 16:01
    jondubois closed #547
  • 16:01
    jondubois commented #547
  • 15:59
    jondubois commented #548
  • 15:59
    jondubois closed #548
  • 15:58
    jondubois closed #549
  • 15:58
    jondubois commented #549
  • 15:55
    jondubois closed #550
  • 15:55
    jondubois commented #550
  • 15:49
    jondubois commented #551
  • 15:48
    jondubois commented #551
Jonathan Gros-Dubois
@jondubois
the 10K figure (per scc-worker) could also be many times higher if the workload is very lightweight
it assumes a message going out to every single user at least every minute or so
but if most connections are kind of indle
then it could be closer to 100K or even 250K per server
May even be possible to reach close to 1M with a lot of optimizations
so it varies a lot
Jonathan Gros-Dubois
@jondubois
@04116 On the scc-broker host, you can tell the scc-broker instance its own IP using the SCC_INSTANCE_IP environment variable; then it will report this IP when connecting to the scc-state server (instead of scc-state trying to detect is automatically)
@04116 an alternative solution (assuming that your haproxy server adds some kind of 'forwarded for' header) is you can tell scc-state what that header is so that it will the value inside that header as the IP. In this case you would need to specify the name of that header as a FORWARDED_FOR_HEADER environment variable on the scc-state host - The value should be the name of the header.
Jonathan Gros-Dubois
@jondubois
@makaziev There have been a few over the years. Most are commercial so you can't see the code. I know that one of our community members is working on an open source one right now though (in progress): https://github.com/johancoppens/simple-chat
@anuragvohraec the socketcluster deployment commands use docker commands behind the scenes so you should be able to set the registry by fiddling with docker CLI settings or environment variables. See https://github.com/SocketCluster/socketcluster/blob/06150056ed688de2ea59ceb2acc6875280dba7f4/bin/cli.js#L600
Jonathan Gros-Dubois
@jondubois
@anuragvohraec I haven't tried to do this before. If it's not possible to set the registry with environment variables, we may have to add an extra argument to the docker login command here https://github.com/SocketCluster/socketcluster/blob/06150056ed688de2ea59ceb2acc6875280dba7f4/bin/cli.js#L585 (feel free to make a pull request).
You may also need to modify your container image names inside the socketcluster-k8s.json config file to include your custom registry ip:port at the start
Jonathan Gros-Dubois
@jondubois
@anuragvohraec About the SSL key, I haven't seen this error before. The SC CLI uses the kubectl create secret tls ${secretName} --key ${privateKeyPath} --cert ${certFilePath}command behind the scenes to add TLS secrets to the K8S cluster. You can try fiddling with that command directly to try to figure out what's wrong.
Oussama Mubarak
@semiaddict
Hi, I'm using SC for the second time, and loving it!
I have a small technical question: how can I use an AGClientSocket listener without assigning the return value to a variable? eslint is throwing the error "'event' is assigned a value but never used" when using the following:
(async () => { for await (const event of socket.listener("connect")) { // Do something without using the event variable } })();
I believe this is more general question for "for..of" and "for await...of" loops, but I can't seem to find the answer any where.
Thanks in advance.
Jonathan Gros-Dubois
@jondubois
@semiaddict it looks like an issue with ESLint. It was raised here eslint/eslint#2342 the issue appears to have been solved for for ... in loops here: eslint/eslint#6126 not sure about for ... of loop (though they were discussed in the issue too). Would ugrading to a newer ESLint version fix this? Or else you can always add an ESLint comment above that line to ignore the rule.
Others also remarked that this rule doesn't make sense for for-in and for-of loops because a variable MUST always be provided so it this rule should not be enforced
Oussama Mubarak
@semiaddict
Thanks @jondubois. I have the latest version of eslint. So I guess I will simply disable it on those lines.
Maurits Lamers
@mauritslamers
Hi all, I have a issue that I don't entirely understand. I have a server with a procedure handler. The client calls this procedure. In the handler on the server I want to immediately after I ended that procedure (through request.end()) call a procedure on the client (through invoke())to get some extra info. However, that call to invoke on the client socket always times out.
Maurits Lamers
@mauritslamers
Small code example:
// server
(async () => {
  for await (let req of socket.procedure('hello')) {
    req.end('hello back!');
    const response = await socket.invoke('what_is_it?'); // will time out
    // do something with response
  }
)();

// client
(async () => {
  for await (let req of socket.procedure('what_is_it?')) {
    req.end('Oh... nothing, just curious...');
  }
})()

(async () => {
   const resp = await socket.invoke('hello')
   // do something with resp
})()
as indicated in the comment, the server will time out waiting for the response of socket.invoke('what_is_it?'), where the client will complain after the restart of the server that the response was already sent
Maurits Lamers
@mauritslamers
The only way I have found until now to make this work, is to wrap the consequent call to invoke in setTimeout. process.nextTick doesn't solve the problem
Maurits Lamers
@mauritslamers
or is the problem that I have created procedures on a client?
Felipe Schulz
@dev_schulz_twitter
hey guys, is there any way to save user information through the middleware?
i'd like to add variables to the users socket object using the middleware
Oussama Mubarak
@semiaddict
Hello @dev_schulz_twitter, I believe you can't add arbitrary data to a socket. You need to keep track of custom data in a seperate object (possibly a Map) and associate each entry with the socket id (making sure to remove entries when sockets disconnect). You can also store some data in the authToken if using authentication, but I don't know if this is considered good practice, as I think the data is also sent to the client side as a JWT token.
Brian Parbhu
@bparbhu
Hey guys, I'm just getting started with using socketcluster and something I'm curious about is the python client. Is there a more updated client for python or is this repo https://github.com/sacOO7/socketcluster-client-python the only trusted one?
Dinh Quang Bui
@bquangDinh
Hi guys, just took a look at this amazing project. I've been looking for a way to implement a concurrent server for our game server where each room lies on one core. And SocketCluster just caught my eyes. I am wondering how does SocketCluster work in general? Is there any diagram or something similar? I've been reading the documentation but there isn't much. It has broker and worker, but I don't really know how do they work together.
Paul
@wilensky

Hi Guys! It seems I wasn't here for a while ) I recently faced a mystery (an issue) while tried to load test my implementation and here what I've got:

  • I see both HANDSHAKE_WS and HANDSHAKE_SC successfully with .allow()
  • I do receive an event in the stream which is server.listener('handshake') and it finishes ok as well
  • ... BUT ...
  • Next I expect connect to appear on socket itself to provide some logic BUT ITS NOT

and this appears only on some certain number of concurrent connections (locally currently) and this number is not that big to tell that it should start suffer.

@SloppyShovel @jondubois any ideas ? I suppose even if there is a backpressure it should resolve this build-up anyway without getting stuck-dead. Middlewares actually don't call any APIs etc. its all native attaches until connect, but it stops to appear.

Paul
@wilensky
  • In case everything is ok server.listener('handshake') triggers first then HANDSHAKE_SC.
  • In case its stuck HANDSHAKE_SC goes first and then server.listener('handshake') is triggered, but no connect on socket.
Paul
@wilensky
image.png

@bquangDinh SC is a cluster implementation, so there is a

  • State instance which knows everything about who is currently up (workers and brokers)
  • Workers are actual socket servers which should be behind a load balancer. Workers are not connected to each other directly.
  • Brokers are channel routers. They know which worker listens which channels and smart enough to route messages to proper workers (not just broadcast).

Communication is built on channels (what is pretty good abstraction and decoupling mechanism if treated properly). If you want to transmit smth. from one socket to another - you use a channel rather than attempting to put smth. directly to an unknown socket.

I shared my diagram above (at least this is AFAI understood it long time ago :))

Paul
@wilensky
And we actually run it in ECS. Not K8s.
Oussama Mubarak
@semiaddict
Hi @wilensky, what do you mean by "this number is not that big"? How many concurrent connections are there exactly?
I am asking as I faced a similar issue a couple of months ago, and it appeared to be a bottleneck in my local network, so it had nothing to do with SC: my local equipment simply didn't support the high number of connections. The issue did not appear on production servers.
Paul
@wilensky
@semiaddict literally it is local benchmark with 20 threads (1 request each) which is 20 concurrent requests. SCC is running in cluster mode in Docker aaanddd ... its randomly stuck on some connections not going beyond handshake. With less number e.g. 10 15 it can be blazing fast. But 20 consistently fails.
Jonathan Gros-Dubois
@jondubois
@wilensky 20 concurrent requests is tiny. I've tested SC many times with 20K concurrent sockets spamming the server every couple of seconds. It could be that there is a condition in your code which means that action.allow() is not being called inside the middleware and so the requests never proceed to the next stage.
Or there is something wrong with the way you're handling the 'connection' stream.
You can have a look at https://socketcluster.io/docs/basic-usage/ and https://socketcluster.io/docs/middleware-and-authorization/ to make sure you're doing everything correctly
Also make sure that connections are actually happening in parallel
For example
(async () => {
  for await (let {socket} of agServer.listener('connection')) {

    (async () => {
      // Set up a loop to handle remote transmitted events.
      for await (let data of socket.receiver('customRemoteEvent')) {
        // ...
      }
    })();

  }
})();
Not
(async () => {
  for await (let {socket} of agServer.listener('connection')) {

      // Set up a loop to handle remote transmitted events.
      for await (let data of socket.receiver('customRemoteEvent')) {
        // ...
      }

  }
})();
The first one allows connections to happen in parallel
Jonathan Gros-Dubois
@jondubois
the second snippet is incorrect. The receiver stream is serial with the main connection handling stream.
so the server will wait for that 'customRemoteEvent' stream on the previous socket to end before it will handle the next socket... Obviously not what you'd typically want.
Paul
@wilensky

Yes, @jondubois this is how we have it implemented on base level, literally the code is as follows :point_down: and its generic for listening each stream.

:information_source: We use Continuation Local Storage for each event

for await (const event of stream) {
      ClsNs.run(() => {
             // .... execution goes here and there
      });
}

So it doesn't seem to block processing of next event in the stream.

Another question that I'm trying to understand is handshake. Does HANDSHAKE mw uses same stream as server.listener('handshake') or mw goes earlier :question:

Paul
@wilensky
Actually this is the only early difference I see, bec. order of those 2 swaps in bad case.
Paul
@wilensky
So, the case is that connect event actually coming, but we are missing it bec. we are not yet listening the stream in this particular moment and we are not .createConsumable() eventually.
Charles
@charuru:matrix.org
[m]
How do I install with kubernetes on digitalocean?
This is what I see on DO:
Oussama Mubarak
@semiaddict
Hello. I'm trying to figure out the best way to handle auth changes on multiple browser tabs/windows.
When a client is authenticated or unauthenticated on one tab, I would like it to do the save on all other tabs.
Since the localStorage is shared by all tabs on the same domain, I was thinking of listening to storage events, but I'm not sure this is the best solution.
Is there a built in way to deal with this? If not, would it make sense to have it built in (as an option, of course)?
I don't mind spending some time on this if it makes sense for the community.