scalecube-js/Lobby

Scott Morris

@smorrisfv_gitlab

I'll see what I can do, it's not personal code, so I'm not sure if I can. I don't have a simple example atm

Idan Levin

@idanilt

try running something simple like this: https://github.com/scalecube/scalecube-js/blob/develop/packages/examples/node/index.js

Scott Morris

@smorrisfv_gitlab

let me see what I can do with that, maybe I can make it deployable in our setup, similar to how I have it setup for my actual implementation

Ronen

@ronenhamias

when nodes publishing them self to the seed cluster trying to access one another

they need to use the host:port that is external to the pod

as they are given by k8s

Scott Morris

@smorrisfv_gitlab

everything is currently all in the same cluster and are using serviceName:port

Ronen

@ronenhamias

when they are sending the request can you see what host and port they are actually trying to connect to?

Scott Morris

@smorrisfv_gitlab

not directly from the command line, but you just gave me an idea, I'll setup tcpdump in the proxy pod and see what it is sending 🤔. Give me a sec!

Ronen

@ronenhamias

cool

Scott Morris

@smorrisfv_gitlab

are proxy requests set over UDP at all? or just TCP?

Idan Levin

@idanilt

TCP
you can work with raw TCP or WS

I am creating a branch with better example and k8s

Scott Morris

@smorrisfv_gitlab

that's cool. I'm analyzing the packet capture now

I'm using TCP and using the Address object to set the address, { "protocol": "tcp","host": "production-auto-deploy.api-250-production","port": 5000,"path": "" }.

Idan Levin

@idanilt

I miss lead you, the service call is TCP, the discovery is UDP

Scott Morris

@smorrisfv_gitlab

I'm using the same JSON object as the seed address, from the packet capture it appears there is some error with the UDP connection, so maybe that has something to do with it

@idanilt

I miss lead you, the service call is TCP, the discovery is UDP

I'm looking into what's up with the discovery atm

Idan Levin

@idanilt

{ "protocol": "tcp","host": "production-auto-deploy.api-250-production","port": 5000,"path": "" }
You config seem ok, it will send service calls via TCP and discovery events via UDP

Scott Morris

@smorrisfv_gitlab

well, i broke it, hah. My seed MS was using { "protocol": "tcp","host": "0.0.0.0","port": 5000,"path": "" } so that the service would listen for all requests. This works in my docker-compose setup, but apparently breaks the UDP service discovery (as it isn't a valid hostname, it resolves to 127.0.0.1). When I changed it to the one that I shared a few messages back it breaks the TCP server.

using netstat with the proper hostname I get,

/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node

Ronen

@ronenhamias

yhea usually k8s and hostnames are the issue :)

Idan Levin

@idanilt

don't use 0.0.0.0
This is the address that the node will publish to the world
The node will try send 0.0.0.0 messages

Scott Morris

@smorrisfv_gitlab

now i'm curious why my docker-compose setups works, as i'm using hostnames there 🤔

Ronen

@ronenhamias

i think that docker compose wire the network between the dockers somehow

Scott Morris

@smorrisfv_gitlab

Yeah, when I use 0.0.0.0 I can talk to the node from anywhere (expected) but it fails in discovery as it reports that its hostname is 0.0.0.0 and thus resolved to 127.0.0.1

Idan Levin

@idanilt

you can use hostnames

what is this hostname: production-auto-deploy.api-250-production?

it seem like all of your namespace

Scott Morris

@smorrisfv_gitlab

i figured out why the docker one works. The hostname I'm using as part of my docker-compose setup is the hostname that is assigned to the seed pod that I spin up. Since it's its own hostname it binds the TCP service to its own IP address and is happy.

In k8s when I assign the hostname production-auto-deploy.api-250-production that is the hostname of the service that is created that points to the pod that runs the microservice, and thus isn't the hostname of the pod itself. I bet there is an error binding to the IP address that is resolved, since it isn't the pod's IP

Scott Morris

@smorrisfv_gitlab

Looking at the docker container, the UDP service seems to always bind on 0.0.0.0, where as the TCP service binds on the hostname that is provided to it. The UDP service then passes along the provided address regardless if the TCP service binds or not

/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.23.0.2:5000         0.0.0.0:*               LISTEN      24/node
⋮
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node

Idan Levin

@idanilt

It's two separated modules, the service calls and the discovery..
you said in the begging, you have rsocket error, rsocket used only for the service call, so the error in the service call module
It can be because the binding is failing or it's try to call unreachable address

Scott Morris

@smorrisfv_gitlab

i found in my packet capture what the rsocket call is returning, and i'm trying to track what's up with it, it doesn't give me much to go on

actually, maybe it does, RSocket isn't HTTP, so this is something with my application 😀, i believe

maybe it will still return data even if the UDP discovery is broken, since i'm only using one side as a proxy?

Idan Levin

@idanilt

you can use WS, if it will help you

The proxy is not an actual proxy

Whats happens is:

Idan Levin

@idanilt

Every micro-service instance create 2 servers:
First one is discovery (swim/gossip over udo), second one is for service calls (rsocket over tcp or WS)
The nodes connect to the seed and sink up list of servers of "service calls"

When you are creating "proxy" you actually creating a client (with the same methods of the service instance)
When you are calling proxy.someMethod() it will try to connect the rsocket server using the address it's got from the discovery

If you are getting the error on the instance creation it probably binding issue
If it's happens when you are trying to invoke (proxy.someMethod()) it's probably because the address isn't reachable

Ido Katz

@katzIdo

hey, can you try to set debug:true in the microservice container that you use for creating the proxy?
maybe the service is not registered yet

Scott Morris

@smorrisfv_gitlab

both sides have the debug flag on, the seed address is also its own microservice, so it seems to register with itself, but I never see it via the client, as my UDP service discovery is broken due to me trying to get the TCP service to bind

it looks like the UDP can resolve only one way

the UDP can talk once to seed MS, but then when it finds out that the seeds address is 0.0.0.0 it fails, since that resolves to 127.0.0.1

Scott Morris

@smorrisfv_gitlab

I've grabbed my latest tcpdump and followed the TCP stream making the request

Scott Morris

@smorrisfv_gitlab

I am creating a branch with better example and k8s

@idanilt you mentioned you're creating an example for k8s, in your example, how do you propose to wire the services together?

Idan Levin

@idanilt

@smorrisfv_gitlab I didn't had time to work on the example (it almost midnight here, I just started)
I encountered something similar in the past, I don't remember how I wired k8s
Anyway tomorrow I will continue, once I have something I will ping you

Scott Morris

@smorrisfv_gitlab

Thanks, I'm going to post a bit more about my setup with the proxy and the seed/microservice. I hope it makes sense.

Here is a diagram of my infrastructure setup, it appears the Proxy Pod communicates with the Microservice Pod [MSP] Service but it gets stopped there (?). The X represents the data isn't getting through to the MSP. I've used tcpdump on both the Proxy Pod and the MSP and the Proxy Pod makes a request via TCP but it fails, the MSP never receives the request. The UDP traffic seems to work until it gets back the MSP hostname (which I have set to 0.0.0.0 to get it to bind the TCP service.)

Proxy Pod* ══Request (TCP)══⇒ Service [MSP]† ════X════⇒ Microservice Pod‡ [MSP] (Seed)
 4000/TCP                       5000/TCP                  5000/TCP
 4000/UDP                       5000/UDP                  5000/UDP

* Proxy Pod
Given the service address of its K8s service, just like the MSP. It's diagram is the inverse TCP flow of the one above.

† MSP Service
Service front end for the MSP that exposes a consistant hostname that will mapp to the MSP regardless if its pod changes names, etc.

‡ MSP
Pod hosting the Microservice and acts as the seed pod. Interestingly the UDP traffic between both the MSP and the Proxy Pod seems to work, other than the wrong host name on the MSP end.

Where communities thrive

People

Repo info

Activity