Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 01 16:19

    idanilt on sc-cli-tool

    wip (compare)

  • Jul 06 02:04
    dependabot[bot] labeled #308
  • Jul 06 02:04
    dependabot[bot] opened #308
  • Jul 06 02:04

    dependabot[bot] on npm_and_yarn

    Bump parse-url from 6.0.0 to 6.… (compare)

  • Jun 08 05:50
    zhou-hao opened #827
  • Apr 27 17:21
    dependabot[bot] labeled #307
  • Apr 27 17:21
    dependabot[bot] opened #307
  • Apr 27 17:21

    dependabot[bot] on npm_and_yarn

    Bump async from 2.6.3 to 2.6.4 … (compare)

  • Apr 27 17:19

    dependabot[bot] on npm_and_yarn

    (compare)

  • Apr 27 17:19

    idanilt on develop

    Bump minimist from 1.2.5 to 1.2… (compare)

  • Apr 27 17:19
    idanilt closed #306
  • Apr 05 07:51
    artem-v review_requested #826
  • Mar 26 18:04
    dependabot[bot] labeled #306
  • Mar 26 18:04
    dependabot[bot] opened #306
  • Mar 26 18:04

    dependabot[bot] on npm_and_yarn

    Bump minimist from 1.2.5 to 1.2… (compare)

  • Mar 11 16:33

    idanilt on develop

    Upgrade RSocket (#290) * Upgra… (compare)

  • Mar 11 16:33
    idanilt closed #290
  • Mar 11 16:33
    idanilt closed #269
  • Mar 11 16:26
    idanilt synchronize #290
  • Mar 11 16:26

    idanilt on rsocket-upgrade

    version (compare)

Idan Levin
@idanilt
I miss lead you, the service call is TCP, the discovery is UDP
Scott Morris
@smorrisfv_gitlab
I'm using the same JSON object as the seed address, from the packet capture it appears there is some error with the UDP connection, so maybe that has something to do with it

@idanilt

I miss lead you, the service call is TCP, the discovery is UDP

I'm looking into what's up with the discovery atm

Idan Levin
@idanilt
{ "protocol": "tcp","host": "production-auto-deploy.api-250-production","port": 5000,"path": "" }
You config seem ok, it will send service calls via TCP and discovery events via UDP
Scott Morris
@smorrisfv_gitlab
well, i broke it, hah. My seed MS was using { "protocol": "tcp","host": "0.0.0.0","port": 5000,"path": "" } so that the service would listen for all requests. This works in my docker-compose setup, but apparently breaks the UDP service discovery (as it isn't a valid hostname, it resolves to 127.0.0.1). When I changed it to the one that I shared a few messages back it breaks the TCP server.
using netstat with the proper hostname I get,
/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node
Ronen
@ronenhamias
yhea usually k8s and hostnames are the issue :)
Idan Levin
@idanilt
don't use 0.0.0.0
This is the address that the node will publish to the world
The node will try send 0.0.0.0 messages
Scott Morris
@smorrisfv_gitlab
now i'm curious why my docker-compose setups works, as i'm using hostnames there 🤔
Ronen
@ronenhamias
i think that docker compose wire the network between the dockers somehow
Scott Morris
@smorrisfv_gitlab
Yeah, when I use 0.0.0.0 I can talk to the node from anywhere (expected) but it fails in discovery as it reports that its hostname is 0.0.0.0 and thus resolved to 127.0.0.1
Idan Levin
@idanilt
you can use hostnames
what is this hostname: production-auto-deploy.api-250-production?
it seem like all of your namespace
Scott Morris
@smorrisfv_gitlab

i figured out why the docker one works. The hostname I'm using as part of my docker-compose setup is the hostname that is assigned to the seed pod that I spin up. Since it's its own hostname it binds the TCP service to its own IP address and is happy.

In k8s when I assign the hostname production-auto-deploy.api-250-production that is the hostname of the service that is created that points to the pod that runs the microservice, and thus isn't the hostname of the pod itself. I bet there is an error binding to the IP address that is resolved, since it isn't the pod's IP

Scott Morris
@smorrisfv_gitlab
Looking at the docker container, the UDP service seems to always bind on 0.0.0.0, where as the TCP service binds on the hostname that is provided to it. The UDP service then passes along the provided address regardless if the TCP service binds or not
/usr/src/app # netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.23.0.2:5000         0.0.0.0:*               LISTEN      24/node
⋮
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           24/node
Idan Levin
@idanilt
It's two separated modules, the service calls and the discovery..
you said in the begging, you have rsocket error, rsocket used only for the service call, so the error in the service call module
It can be because the binding is failing or it's try to call unreachable address
Scott Morris
@smorrisfv_gitlab
image.png
i found in my packet capture what the rsocket call is returning, and i'm trying to track what's up with it, it doesn't give me much to go on
actually, maybe it does, RSocket isn't HTTP, so this is something with my application 😀, i believe
maybe it will still return data even if the UDP discovery is broken, since i'm only using one side as a proxy?
Idan Levin
@idanilt
you can use WS, if it will help you
The proxy is not an actual proxy
Whats happens is:
Idan Levin
@idanilt

Every micro-service instance create 2 servers:
First one is discovery (swim/gossip over udo), second one is for service calls (rsocket over tcp or WS)
The nodes connect to the seed and sink up list of servers of "service calls"

When you are creating "proxy" you actually creating a client (with the same methods of the service instance)
When you are calling proxy.someMethod() it will try to connect the rsocket server using the address it's got from the discovery

If you are getting the error on the instance creation it probably binding issue
If it's happens when you are trying to invoke (proxy.someMethod()) it's probably because the address isn't reachable
Ido Katz
@katzIdo
hey, can you try to set debug:true in the microservice container that you use for creating the proxy?
maybe the service is not registered yet
Scott Morris
@smorrisfv_gitlab
both sides have the debug flag on, the seed address is also its own microservice, so it seems to register with itself, but I never see it via the client, as my UDP service discovery is broken due to me trying to get the TCP service to bind
it looks like the UDP can resolve only one way
the UDP can talk once to seed MS, but then when it finds out that the seeds address is 0.0.0.0 it fails, since that resolves to 127.0.0.1
Scott Morris
@smorrisfv_gitlab
I've grabbed my latest tcpdump and followed the TCP stream making the request
Scott Morris
@smorrisfv_gitlab

I am creating a branch with better example and k8s

@idanilt you mentioned you're creating an example for k8s, in your example, how do you propose to wire the services together?

Idan Levin
@idanilt
@smorrisfv_gitlab I didn't had time to work on the example (it almost midnight here, I just started)
I encountered something similar in the past, I don't remember how I wired k8s
Anyway tomorrow I will continue, once I have something I will ping you
Scott Morris
@smorrisfv_gitlab
Thanks, I'm going to post a bit more about my setup with the proxy and the seed/microservice. I hope it makes sense.

Here is a diagram of my infrastructure setup, it appears the Proxy Pod communicates with the Microservice Pod [MSP] Service but it gets stopped there (?). The X represents the data isn't getting through to the MSP. I've used tcpdump on both the Proxy Pod and the MSP and the Proxy Pod makes a request via TCP but it fails, the MSP never receives the request. The UDP traffic seems to work until it gets back the MSP hostname (which I have set to 0.0.0.0 to get it to bind the TCP service.)

Proxy Pod* ══Request (TCP)══⇒ Service [MSP]† ════X════⇒ Microservice Pod‡ [MSP] (Seed)
 4000/TCP                       5000/TCP                  5000/TCP
 4000/UDP                       5000/UDP                  5000/UDP

* Proxy Pod
Given the service address of its K8s service, just like the MSP. It's diagram is the inverse TCP flow of the one above.

† MSP Service
Service front end for the MSP that exposes a consistant hostname that will mapp to the MSP regardless if its pod changes names, etc.

‡ MSP
Pod hosting the Microservice and acts as the seed pod. Interestingly the UDP traffic between both the MSP and the Proxy Pod seems to work, other than the wrong host name on the MSP end.

Idan Levin
@idanilt
I will check tomorrow,signing off for today
Scott Morris
@smorrisfv_gitlab
thanks for looking into this
Idan Levin
@idanilt
Hey @smorrisfv_gitlab
Here k8s example: scalecube/scalecube-js#244
idan@idan-XPS-13-9380:~/prj/scalecube-js/packages/examples/k8s$ kubectl -n scalecube-example logs pod/consumer
seed seed.seed
address 10.244.0.35
******** address: ws://10.244.0.35:8080/********
microservice received an updated
REGISTERED: "HelloService/hello"
hello
hello
hello
hello
hello
hello
idan@idan-XPS-13-9380:~/prj/scalecube-js/packages/examples/k8s$ kubectl -n scalecube-example logs pod/seed
seed undefined
address 10.244.0.36
******** address: ws://10.244.0.36:8080/********
microservice received an updated
REGISTERED: "HelloService/hello"
idan@idan-XPS-13-9380:~/prj/scalecube-js/packages/examples/k8s$ kubectl -n scalecube-example logs pod/greeting
seed seed.seed
address 10.244.0.37
Idan Levin
@idanilt

few tips:

  • use pod address for ms address (also for seed ms)
  • I used DNS for the seed, it's the most simple way, not require from k8s any proxies

If you want you can PR similar setup to yours to my branch and I will check it out (it's hard debug without seeing)

Scott Morris
@smorrisfv_gitlab
Hey @idanilt , Thanks for sharing, I'm working on going through your example and adjusting my setup atm. Thanks for sharing. I'll let you know what happens
Idan Levin
@idanilt
Happy to share
Waiting to hear what happened
Scott Morris
@smorrisfv_gitlab
It looks like you're using a hostname for the seed, seed.seed, and an IP address for the client,status.podIP?
Scott Morris
@smorrisfv_gitlab
I'm trying to figure out how the seed.seed name works, is this autoresolved by k8s' DNS?
Scott Morris
@smorrisfv_gitlab
After some research and digging I discovered using the pod's DNS entry won't work for my setup without significant changes. I'm using a slightly modified Helm chart based on GitLab's Auto Deploy Helm Chart. My setup shims in the ability to map ports but most of the management is left to the chart's defaults. I could, in theory, use env vars to pass the hostname to my TS code, but the actual name won't be known until deployment time, which maybe I could take advantage of in the project itself, but it is dynamic and passing it outside of the project isn't very easy.
Idan Levin
@idanilt

Hey, I can't really talk today
We can talk tomorrow

In general the scalecube advantage that you need only one address for the seed
you can also do static IP or use configMap/secret/shared volume to share the ip
If you can resolve in real time you can use init container (like me) to find the address and set it

Tomorrow, on you early morning, it is the hours I am mostly available

Idan Levin
@idanilt
Hey @smorrisfv_gitlab
Did you managed to make it work?