scalecube-js/Lobby

Scott Morris

@smorrisfv_gitlab

@idanilt I have. I've setup a Microservice in one pod, and setup a microservice with a proxy for the other microservice in another pod. I have opened up TCP and UDP traffic on both pods and via their respective service, and have assigned them their service hostnames/port accordingly. However, when I try and have proxy MS try and call the remote function I get the generic error of RemoteCall exception occur.

i've concluded that the proxy and the MS aren't communicating and am currently trying to get the original error, before it is masked by the RemoteCall exception occur. generic error in the rsocket-adapter code

Idan Levin

@idanilt

Can you share with me the code you trying to run?

Scott Morris

@smorrisfv_gitlab

I'll see what I can do, it's not personal code, so I'm not sure if I can. I don't have a simple example atm

Idan Levin

@idanilt

try running something simple like this: https://github.com/scalecube/scalecube-js/blob/develop/packages/examples/node/index.js

Scott Morris

@smorrisfv_gitlab

let me see what I can do with that, maybe I can make it deployable in our setup, similar to how I have it setup for my actual implementation

Ronen

@ronenhamias

when nodes publishing them self to the seed cluster trying to access one another

they need to use the host:port that is external to the pod

as they are given by k8s

Scott Morris

@smorrisfv_gitlab

everything is currently all in the same cluster and are using serviceName:port

Ronen

@ronenhamias

when they are sending the request can you see what host and port they are actually trying to connect to?

Scott Morris

@smorrisfv_gitlab

not directly from the command line, but you just gave me an idea, I'll setup tcpdump in the proxy pod and see what it is sending 🤔. Give me a sec!

Ronen

@ronenhamias

cool

Scott Morris

@smorrisfv_gitlab

are proxy requests set over UDP at all? or just TCP?

Idan Levin

@idanilt

TCP
you can work with raw TCP or WS

I am creating a branch with better example and k8s

Scott Morris

@smorrisfv_gitlab

that's cool. I'm analyzing the packet capture now

I'm using TCP and using the Address object to set the address, { "protocol": "tcp","host": "production-auto-deploy.api-250-production","port": 5000,"path": "" }.

Idan Levin

@idanilt

I miss lead you, the service call is TCP, the discovery is UDP

Scott Morris

@smorrisfv_gitlab

I'm using the same JSON object as the seed address, from the packet capture it appears there is some error with the UDP connection, so maybe that has something to do with it

@idanilt

I miss lead you, the service call is TCP, the discovery is UDP

I'm looking into what's up with the discovery atm

Idan Levin

@idanilt

{ "protocol": "tcp","host": "production-auto-deploy.api-250-production","port": 5000,"path": "" }
You config seem ok, it will send service calls via TCP and discovery events via UDP

Scott Morris

@smorrisfv_gitlab

well, i broke it, hah. My seed MS was using { "protocol": "tcp","host": "0.0.0.0","port": 5000,"path": "" } so that the service would listen for all requests. This works in my docker-compose setup, but apparently breaks the UDP service discovery (as it isn't a valid hostname, it resolves to 127.0.0.1). When I changed it to the one that I shared a few messages back it breaks the TCP server.

using netstat with the proper hostname I get,

/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node

Ronen

@ronenhamias

yhea usually k8s and hostnames are the issue :)

Idan Levin

@idanilt

don't use 0.0.0.0
This is the address that the node will publish to the world
The node will try send 0.0.0.0 messages

Scott Morris

@smorrisfv_gitlab

now i'm curious why my docker-compose setups works, as i'm using hostnames there 🤔

Ronen

@ronenhamias

i think that docker compose wire the network between the dockers somehow

Scott Morris

@smorrisfv_gitlab

Yeah, when I use 0.0.0.0 I can talk to the node from anywhere (expected) but it fails in discovery as it reports that its hostname is 0.0.0.0 and thus resolved to 127.0.0.1

Idan Levin

@idanilt

you can use hostnames

what is this hostname: production-auto-deploy.api-250-production?

it seem like all of your namespace

Scott Morris

@smorrisfv_gitlab

i figured out why the docker one works. The hostname I'm using as part of my docker-compose setup is the hostname that is assigned to the seed pod that I spin up. Since it's its own hostname it binds the TCP service to its own IP address and is happy.

In k8s when I assign the hostname production-auto-deploy.api-250-production that is the hostname of the service that is created that points to the pod that runs the microservice, and thus isn't the hostname of the pod itself. I bet there is an error binding to the IP address that is resolved, since it isn't the pod's IP

Scott Morris

@smorrisfv_gitlab

Looking at the docker container, the UDP service seems to always bind on 0.0.0.0, where as the TCP service binds on the hostname that is provided to it. The UDP service then passes along the provided address regardless if the TCP service binds or not

/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.23.0.2:5000         0.0.0.0:*               LISTEN      24/node
⋮
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node

Idan Levin

@idanilt

It's two separated modules, the service calls and the discovery..
you said in the begging, you have rsocket error, rsocket used only for the service call, so the error in the service call module
It can be because the binding is failing or it's try to call unreachable address

Scott Morris

@smorrisfv_gitlab

i found in my packet capture what the rsocket call is returning, and i'm trying to track what's up with it, it doesn't give me much to go on

actually, maybe it does, RSocket isn't HTTP, so this is something with my application 😀, i believe

maybe it will still return data even if the UDP discovery is broken, since i'm only using one side as a proxy?

Idan Levin

@idanilt

you can use WS, if it will help you

The proxy is not an actual proxy

Whats happens is:

Idan Levin

@idanilt

Every micro-service instance create 2 servers:
First one is discovery (swim/gossip over udo), second one is for service calls (rsocket over tcp or WS)
The nodes connect to the seed and sink up list of servers of "service calls"

When you are creating "proxy" you actually creating a client (with the same methods of the service instance)
When you are calling proxy.someMethod() it will try to connect the rsocket server using the address it's got from the discovery

If you are getting the error on the instance creation it probably binding issue
If it's happens when you are trying to invoke (proxy.someMethod()) it's probably because the address isn't reachable

Ido Katz

@katzIdo

hey, can you try to set debug:true in the microservice container that you use for creating the proxy?
maybe the service is not registered yet

Scott Morris

@smorrisfv_gitlab

both sides have the debug flag on, the seed address is also its own microservice, so it seems to register with itself, but I never see it via the client, as my UDP service discovery is broken due to me trying to get the TCP service to bind

it looks like the UDP can resolve only one way

the UDP can talk once to seed MS, but then when it finds out that the seeds address is 0.0.0.0 it fails, since that resolves to 127.0.0.1

Scott Morris

@smorrisfv_gitlab

I've grabbed my latest tcpdump and followed the TCP stream making the request

Scott Morris

@smorrisfv_gitlab

I am creating a branch with better example and k8s

@idanilt you mentioned you're creating an example for k8s, in your example, how do you propose to wire the services together?

Where communities thrive

People

Repo info

Activity