jenkinsci/remoting

Jenkins Remoting and agents

Oleg Nenashev

@oleg-nenashev

@afalko @pvtuan10 are you fine with shipping jenkins-infra/jenkins.io#2408 ? There is no approvals from you in the blogpost

Oleg Nenashev

@oleg-nenashev

@/all https://jenkins.io/blog/2019/08/19/remoting-kafka-kubernetes-release-2/ - new blogpost from @longngn

@longngn ^ would be great to add a lnk to the plugin in the beginning of the blogpost

Nguyen Le Vu Long

@longngn

@oleg-nenashev Do you have any idea what ci/dockercloud check might be in PR: jenkinsci/remoting-kafka-plugin#77 ?

Andrey Falko

@afalko

@oleg-nenashev : Do you have access to view error from dockerhub CI? jenkinsci/remoting-kafka-plugin#77

we don't have access to view the error and fix it

Andrey Falko

@afalko

https://cloud.docker.com/redirect/?resource_uri=/api/audit/v1/action/531504c9-1aec-4ecc-9d69-efcca3340334/

Andrey Falko

@afalko

Looks like it runs in the master branch too :/

Bogdan Suta

@bogdansuta-intel

Hello guys and gals, I have an issue with the Remoting endpoint of Jenkins (running, by default, on port 50000) crashing and not accepting new connections (curling the URL just hangs). The only way the situation recovers is just by restarting Jenkins. Any ideas? Can somebody guide me in the right direction?

Or is there a way to restart the component from the Jenkins scripting console?

Martin d'Anjou

@martinda

Hello @pvtuan10 @afalko Please complete the GSoC evaluation as soon as possible. There are only 3 days left and it is a long weekend in North America.

Andrey Falko

@afalko

@martinda : I'm working on it!

Martin d'Anjou

@martinda

@afalko thank you.

MEscribano

@mescribano

Hello everyone,
I have an issue with the Remoting Kafka Plugin on my Jenkins instance (Jenkins ver. 2.176.2).
I have installed the plugin in the Jenkins-Master and the connection to the zookeeper and kafka are sucessfull.

I created manually a node and the launch ist not working, it says "This agent is offline because Jenkis failed to lauch the agent process on it".
I have downloaded the JAR file from here https://repo.jenkins-ci.org/releases/io/jenkins/plugins/remoting-kafka/remoting-kafka-agent/, Version 2.0.1 and I have tried to execute the JAR from my local machine
java -jar remoting-kafka-agent.jar -name node-name -master http://jenkins.url/ -secret XXYY -kafkaURL XXX.YY.ZZZ.XXX:PPPP -noauth
However this doesn't work either. The console output says at the end

[kafka-producer-network-thread | producer-2] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Connection to node -1 (/XXX.YY.ZZZ.XXX:PPPP) could not be established. Broker may not be available.
[Channel reader thread: node-name] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=jenkins.url.-1.node-name-id] Connection to node -1 (/XXX.YY.ZZZ.XXX:PPPP) could not be established. Broker may not be available.

I don't find anything in the logs of kafka and also not in the Sys logs of jenkins Master.

Do you have an idea how can I approach the poblem? Thanks. María

Nguyen Le Vu Long

@longngn

@mescribano it looks like problems with Kafka config, did you set Kafka’s advertised listener correctly?

MEscribano

@mescribano

@longngn: thanks for the fast reply. For the Kafka I have used the Docker-Image (confluent/kafka:latest)
the server.properties has no listeners property.
I believe I am missing the properties KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS as env variables in my statefulset yaml.
Which value should have these two properties in my case?

Nguyen Le Vu Long

@longngn

Can you try setting KAFKA_ADVERTISED_LISTENERS to PLAINTEXT://XXX.YY.ZZZ.XXX:PPPP (where XXX.YY.ZZZ.XXX:PPPP is the Kafka address Jenkins use to connect)

then you can try to connect to Kafka using kafkacat from the same host as Jenkins master or agent

more details: https://rmoff.net/2018/08/02/kafka-listeners-explained/

MEscribano

@mescribano

thanks @longngn, we will try that !

Oleg Nenashev

@oleg-nenashev

Thanks for your reviews @res0nance !

Raihaan Shouhell

@res0nance

:+1:

MEscribano

@mescribano

Hello @longngn,
I have set both properties to the following:

        - name: KAFKA_LISTENERS
          value: 'PLAINTEXT://localhost:9092'
        - name: KAFKA_ADVERTISED_LISTENERS
          value: 'PLAINTEXT://<kafka-ip>:9092'

However the same problem still persists (see my comment from Sept 4th).

The configuration of the plugin inside the Jenkins-Master says that connection to the zookeeper and kafka are both successful... So I assume that my Jenkins-Master connects properly to the Kafka.

Can you assist us on this? We are stucked...

Thanks,
María

Nguyen Le Vu Long

@longngn

@mescribano the Test Connection button of the plugin only performs a simple check to Kafka and Zookeeper servers. It doesn't tell whether creating topics or publish/subscribe to topics would also be successful.

If you want to fully test Kafka config, you should SSH into Jenkins master host machine and try to create topic/publish/subscribe to Kafka servers using either kafkacat or Kafka binaries

MEscribano

@mescribano

Hi @longngn, thanks for the fast reply.
I downloaded the kafka binaries and followed this quickstart: https://kafka.apache.org/quickstart
I am able to create a topic and I can see it as well when I run the list topic command. The Jenkins <-> Kafka connection seem to work properly.

When I try to execute the java -jar command from my jenkins-master container I get following error:
[Thread-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=<jenkins-url>.-1.kafka1-id] 1 partitions have leader brokers without a matching listener, including [<jenkins-url>.-1.kafka1-topic-0]

Do you have any idea how can I approach this problem?

Thanks a lot,
María

gjf974

@gjf974

Hello All. I re-send my message here .

I would like to know if we can connect a jenkins slave agent on the https port (443), slave and master are on 2 different networks and all other ports are closed.

Can you help me?
Thanks

Oleg Nenashev

@oleg-nenashev

@gjf974 You would need to setup TCP tunneling over HTTPs to do so. It is doable, but it is not how Remoting works OOTB. There were some discussions about implementing Remoting over websockets, but there is no pending PRs for that so far

Jesper Lehtinen

@jjlehtinen

Hey everyone, any idea why JLNP4 would suddenly stop working? To connect our agents using jlnp, I had to enable jnlp3 as a fallback. Where can I start debugging this?

https://github.com/jenkinsci/remoting/blob/master/docs/troubleshooting.md

Jesper Lehtinen

@jjlehtinen

Connection closed before acknowledgement sent

Jeff Thompson

@jeffret-b

That's a strange one. Usually when some result changes like this, it's because something else somewhere changed. Tracking down that change can be difficult.

It sounds kind of like this issue: https://issues.jenkins-ci.org/browse/JENKINS-57143

Some other experiences here: https://stackoverflow.com/questions/44698782/jenkins-slave-cannot-connect-with-master-incorrect-acknowledgement-sequence

I'd start by looking at the Github OAuth plugin, seeing what version you have and if you can locate any relationship between it and the problem.

Jesper Lehtinen

@jjlehtinen

Thanks, it seems like we are running 0.33 right now, will need to dig deeper

Oleg Nenashev

@oleg-nenashev

Threads have been enabled in this room

Oleg Nenashev

@oleg-nenashev

@/all JNLP1..3 protocols were removed in 3.40, and they will be removed from the next Jenkins weekly: https://groups.google.com/forum/#!topic/jenkinsci-dev/VBeEhaE7VcQ

Also https://twitter.com/oleg_nenashev/status/1216102073357434881

EaswarDSharma

@EaswarDSharma

hello guys, i want to work on the following newbie-friendly issue, can some guide?
https://issues.jenkins-ci.org/browse/JENKINS-49905

4 replies

ZacharyChua

@zachary2940

Hi I'm a student from Singapore and I want to contribute to this project

Is there a place I can start from? I am studying distributed systems this semester so I have some knowledge about remoting.

Oleg Nenashev

@oleg-nenashev

@zachary2940 hi, thanks for the interest! The most simple way to start is to try out Jenkins with distributed masters. There is a newly introduced Websocket support, and you can explore it: https://jenkins.io/blog/2020/02/02/web-socket/ . Adding native support to plugins like Swarm Plugin could be a good first step.

Also, there are some open newbie-friendly issues: https://issues.jenkins-ci.org/browse/JENKINS-40700?jql=labels%20%3D%20newbie-friendly%20and%20status%20in%20(Open%2C%20%22To%20Do%22%2C%20Reopened)%20and%20component%20%20%3D%20remoting

ZacharyChua

@zachary2940

Is anybody working on this now https://issues.jenkins-ci.org/browse/JENKINS-40700? Sorry I have no idea how to add a comment directly to the issue

Raihaan Shouhell

@res0nance

@zachary2940 you would need to make an account on jenkins and sign in, only then you will be able to see the comment button.

Raihaan Shouhell

@res0nance

I'm not sure if this has been ever reported or is a known issue but calling /disconnect or having a computer's availability setting disconnect an agent consistently causes a ClosedChannelException so in essence SlaveComputer.disconnect will cause this exception to happen. This seems to only happen to "JNLP" connections AFAICT. Stacktrace below:

java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:784)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
    at hudson.remoting.Channel.close(Channel.java:1452)
    at hudson.remoting.Channel.close(Channel.java:1405)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:844)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:105)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:753)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Miho

@elva1087

Hi all. I'm trying to connect a WINDOWS 10 as a agent by Java web Start method to a Linux Master. But I'v got a response "provided port:6200 is not reachable". I'v closed all firewall programs and I'm sure that port 6200 is not occupied.Is there anyone encoutered issue like this: 三月 02, 2020 11:12:34 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
信息: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
三月 02, 2020 11:12:36 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
警告: Connection refused: connect
三月 02, 2020 11:12:36 下午 hudson.remoting.jnlp.GuiListener$2 run
严重: http://123.126.106.85:8080/ provided port:6200 is not reachable
java.io.IOException: http://123.126.106.85:8080/ provided port:6200 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314)
at hudson.remoting.Engine.innerRun(Engine.java:690)
at hudson.remoting.Engine.run(Engine.java:518)

Jeff Thompson

@jeffret-b

It's probably a network configuration issue, but it's hard to suggest what the issue might be. I recommend you follow some network configuration and reachability troubleshooting. Use some other tools to try to connect to that port. Try verifying traffic between the computers. Try using a different port. It looks like you're hitting the very basic, simple port connection test and it's failing.

Valentin Delaye

@jonesbusy

Hi! I have open this ticket today about an issue with remoting 4.2.1 and and 4.3 (seems a regression from 4.2) related to websocket : https://issues.jenkins-ci.org/browse/JENKINS-62576

I'm aware that the WS feature is still in beta, but it literally saved me to avoid networking issue (firewall, K8S ingress etc...) :)

Except the previous issue with large payload I didn't had previously any issue with connection or stability. Now I'm the connection is crashing in the middle of the build. I had sadly to rollback to 2.222.3

I've put all the details on the ticket, I can help debug or provide more logs if needed.

Thanks!

Where communities thrive

People

Repo info

Activity