jenkinsci/remoting

Jenkins Remoting and agents

Jeff Thompson

@jeffret-b

I'd start by looking at the Github OAuth plugin, seeing what version you have and if you can locate any relationship between it and the problem.

Jesper Lehtinen

@jjlehtinen

Thanks, it seems like we are running 0.33 right now, will need to dig deeper

Oleg Nenashev

@oleg-nenashev

Threads have been enabled in this room

Oleg Nenashev

@oleg-nenashev

@/all JNLP1..3 protocols were removed in 3.40, and they will be removed from the next Jenkins weekly: https://groups.google.com/forum/#!topic/jenkinsci-dev/VBeEhaE7VcQ

Also https://twitter.com/oleg_nenashev/status/1216102073357434881

EaswarDSharma

@EaswarDSharma

hello guys, i want to work on the following newbie-friendly issue, can some guide?
https://issues.jenkins-ci.org/browse/JENKINS-49905

4 replies

ZacharyChua

@zachary2940

Hi I'm a student from Singapore and I want to contribute to this project

Is there a place I can start from? I am studying distributed systems this semester so I have some knowledge about remoting.

Oleg Nenashev

@oleg-nenashev

@zachary2940 hi, thanks for the interest! The most simple way to start is to try out Jenkins with distributed masters. There is a newly introduced Websocket support, and you can explore it: https://jenkins.io/blog/2020/02/02/web-socket/ . Adding native support to plugins like Swarm Plugin could be a good first step.

Also, there are some open newbie-friendly issues: https://issues.jenkins-ci.org/browse/JENKINS-40700?jql=labels%20%3D%20newbie-friendly%20and%20status%20in%20(Open%2C%20%22To%20Do%22%2C%20Reopened)%20and%20component%20%20%3D%20remoting

ZacharyChua

@zachary2940

Is anybody working on this now https://issues.jenkins-ci.org/browse/JENKINS-40700? Sorry I have no idea how to add a comment directly to the issue

Raihaan Shouhell

@res0nance

@zachary2940 you would need to make an account on jenkins and sign in, only then you will be able to see the comment button.

Raihaan Shouhell

@res0nance

I'm not sure if this has been ever reported or is a known issue but calling /disconnect or having a computer's availability setting disconnect an agent consistently causes a ClosedChannelException so in essence SlaveComputer.disconnect will cause this exception to happen. This seems to only happen to "JNLP" connections AFAICT. Stacktrace below:

java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:784)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
    at hudson.remoting.Channel.close(Channel.java:1452)
    at hudson.remoting.Channel.close(Channel.java:1405)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:844)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:105)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:753)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Miho

@elva1087

Hi all. I'm trying to connect a WINDOWS 10 as a agent by Java web Start method to a Linux Master. But I'v got a response "provided port:6200 is not reachable". I'v closed all firewall programs and I'm sure that port 6200 is not occupied.Is there anyone encoutered issue like this: 三月 02, 2020 11:12:34 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
信息: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
三月 02, 2020 11:12:36 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
警告: Connection refused: connect
三月 02, 2020 11:12:36 下午 hudson.remoting.jnlp.GuiListener$2 run
严重: http://123.126.106.85:8080/ provided port:6200 is not reachable
java.io.IOException: http://123.126.106.85:8080/ provided port:6200 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314)
at hudson.remoting.Engine.innerRun(Engine.java:690)
at hudson.remoting.Engine.run(Engine.java:518)

Jeff Thompson

@jeffret-b

It's probably a network configuration issue, but it's hard to suggest what the issue might be. I recommend you follow some network configuration and reachability troubleshooting. Use some other tools to try to connect to that port. Try verifying traffic between the computers. Try using a different port. It looks like you're hitting the very basic, simple port connection test and it's failing.

Valentin Delaye

@jonesbusy

Hi! I have open this ticket today about an issue with remoting 4.2.1 and and 4.3 (seems a regression from 4.2) related to websocket : https://issues.jenkins-ci.org/browse/JENKINS-62576

I'm aware that the WS feature is still in beta, but it literally saved me to avoid networking issue (firewall, K8S ingress etc...) :)

Except the previous issue with large payload I didn't had previously any issue with connection or stability. Now I'm the connection is crashing in the middle of the build. I had sadly to rollback to 2.222.3

I've put all the details on the ticket, I can help debug or provide more logs if needed.

Thanks!

Balazs Varnai

@bvarnai

Hi! I have nginx proxy setup, using normal JNPL agents. Now I switched (trying) to WebSocket, but receiving a strange error

JNLP file https://xzy.com/jenkins/computer/vsb-10/slave-agent.jnlp?encrypt=true has invalid arguments: [89aacf49720cd4550ead6eb23060e995201842915acddff53776a6baa69177bd, node-10, -webSocket, -workDir, c:\workdir\vsb\jenkins, -internalDir, remoting, -url, https://xyz.com/jenkins/, -url, http://xzy-1.com:8080/jenkins/, -headless, -workDir, c:\workdir\vsb\jenkins, -internalDir, remoting]
Most likely a configuration error in the master
-webSocket supports only a single -url

Any suggestions I should try?
Thanks!

3 replies

Jane

@Mrzhuang007

Hi~ Which Class will the remoting write the build log to Jenkins Master?

1 reply

Zahi

@370672701

Hello ，How to configure Jenkins behind Nginx reverse proxy for JNLP slaves to connect， I have below problem with Jenkins 2.235.5 when connect a docker slave

INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
INFO: Agent discovery successful
Agent address: jenkins.operations.svc.cluster.local
Agent port: 50000
INFO: Handshaking
INFO: Connecting to jenkins.operations.svc.cluster.local:50000
INFO: Trying protocol: JNLP4-connect
Apr 16, 2019 1:29:01 PM org.jenkinsci.remoting.protocol.impl.AckFilterLayer abort
WARNING: [JNLP4-connect connection to jenkins.operations.svc.cluster.local/10.19.251.169:50000] Incorrect acknowledgement sequence, expected 0x000341434b got 0x485454502f
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)....

mayank srivastava

@mighty_1989_twitter

@mighty_1989_twitter
@/all can someone help me to understand what are interfaces or extension points are required to write your own cloud interface
i am looking for something like this

also i am looking to implement dynamic agent provisioning

something in similar terms like kubernetes or ec2

given these plugins are quite difficult to understand

i want to understand easy starting point

jayalakshmia20

@jayalakshmia20

@all - can you help to suggest a stable version of Jenkins LTS which can be used ?

jayalakshmia20

@jayalakshmia20

currently i am using Jenkins LTS 2.249.1 but encountered an issue while connecting windows agent to linux master. The connection closed before acknowledgement sent error.

jayalakshmia20

@jayalakshmia20

aloussayef

@aloussayef

Hello,
I'm running jenkins 2.269 in a docker swarm cluster behind traefik 2 and i'm trying to configure the docker plugin 1.2.1 to autoprovision jnlp agents using the inbound agent docker image https://hub.docker.com/r/jenkins/inbound-agent/
the issue that i'm facing is that tcp connection over port 50000 using traefik is not working see https://issues.jenkins.io/browse/JENKINS-62738 so i had to configure my agents to use the websocket. This works fine with permanent agents. But with the docker plugin i keep getting the following error. any thoughts ?

java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

Jeff Thompson

@jeffret-b

I don't have any good troubleshooting tips. You'll have to do some deep troubleshooting on this configuration. Something isn't working when it tries to complete the exception. It kind of looks like you haven't gotten it configured correctly to use Websockets. The ConnectionRefusalException is related to the (confusingly misnamed) JNLP Remoting protocols. I don't think it's used in the Websockets implementation, though I'm not sure. If you do figure something out, please share so others can build on that.

aloussayef

@aloussayef

Thxs @jeffret-b i'm now able to provision on demand ssh, jnlp and docker in docker agents.
The first issue i had to debug was the use of Websocket in the docker plugin template plugin configuration vs adding and env directive in the dockerfile JENKINS_WEB_SOCKET=true.
this lead me to a better stack trace in jenkins system logs. by this time containers were started but then crash immediately. and that's when i realized that i hit this issue while trying to persist workspaces using volumes jenkinsci/docker-plugin#821 installing the beta-release fixed this and i'm now all set.

dsahithi

@dsahithi

Hello,
I'm running jenkins 2.222.4 in k8's cluster, i'm using swarm plugin to launch windows agent to linux master. I'm getting the following error. any thoughts?
java.io.IOException: provided port:50000 is not reachable

Akihrio Kiuchi

@Aki-7

@oleg-nenashev Hi, I'm Akihiro Kiuchi, thank you for answering my questions on the GSoC Office Hour. I'll rewrite my proposal and post it here in a few days. I'll be happy if you review it.

1 reply

Akihrio Kiuchi

@Aki-7

@oleg-nenashev

Akihrio Kiuchi

@Aki-7

Hi, Mr. Nenashev.
I'm writing a proposal to expose the agent metrics on the "agent" node and I have some questions.
I think we need to think carefully about the way to deliver and execute the programs because the process or thread to collect and expose the agent metrics must be running even when the connection to the controller is lost, but at the same time, we cannot lose control of the process or thread.
Now, I come up with two possible types of the way to execute the programs. One is simply executing them as a thread or subprocess and the other is creating a UNIX/Windows service. The advantage of launching as a service is that we can keep collecting the metrics even if the agent is restarted while the connection to the controller is lost. But I think launching as a thread is better because launching as a service has a critical problem. Thinking about uninstalling this plugin while the connection is lost, we completely lose control of the service while the service is still running. The same thing can be said for launching as a thread but the thread will stop at least when the agent restarts. It is much better, I think. What do you think about this? And if you know a similar project which handles this problem, I want to know it.

Akihrio Kiuchi

@Aki-7

HI, @oleg-nenashev.

I wrote my first proposal draft, and I would like you to review it.

https://docs.google.com/document/d/1SYgJAfAuW9wzaUaAXgKODJ7DFr21y1P61yhjltXmwpc/edit?usp=sharing

Oleg Nenashev

@oleg-nenashev

Thanks @Aki-7 ! I have added a few comments. Will try to provide more feedback over the week. Also, it might make sense to send this proposal to the Jenkins developer mailing list

Akihrio Kiuchi

@Aki-7

Thank you so much! I’ll do it now!

Darshan Kansagara

@darshank15

Hi @Aki-7 Looks good, Also possible to just explore and see how you can use these collected (Prometheus) data to create a nice dashboard for Jenkin's admin. Just think of different monitoring we can do using this plugin, and what data needs to fetch using an agent and then create some charts from it would be good idea. Maybe we can also consider it into the proposed schedule.

1 reply

Oleg Nenashev

@oleg-nenashev

Thanks for sending the proposal to the devlist @Aki-7 !

Oleg Nenashev

@oleg-nenashev

@Aki-7 Hi! If you have some time, it might make sense to take a look at the OpenTelemetry plugin: https://github.com/jenkinsci/opentelemetry-plugin . It also exposes some node/executor metrics in an open format

2 replies

werstatyw

@werstatyw

Hello Oleg, participants and other members of channel. Please find some time and review my proposal https://docs.google.com/document/d/1Mq9PAxUbQ_TUu7u_hEGkgzdUQEhXGiCDoYcxISJdda4/edit

1 reply

This year I'm going to fully commit in k8s, Jenkins and implement security web hooks. https://www.jenkins.io/projects/gsoc/2021/project-ideas/jenkins-operator-security/ The main thing which I puzzled at the moment - did i broke down the deliverables correctly?

Oleg Nenashev

@oleg-nenashev

@/all Jenkins agent user survey! Akihiro Kiuchi, a GSoC 2021 student in Jenkins/CDF, is looking for feedback about Jenkins remoting pain points, and how could we improve agent monitoring and troubleshooting experience with OpenTelemetry. https://docs.google.com/forms/d/e/1FAIpQLSdiuQN3sm2mQ2E86VTXVXu7bf_9C0hVdzhw2_Kvu3DFqL7EZA/viewform

Ridae Hamdani

@ridaeh

hi folks;
Where I can find the jenkins cli websocket logs ?
Thanks in advance

Dor

@dsaydon90

Hi! Anyone here knows how to handle Agent crash issues like "java.io.IOException: The specified network name is no longer available"?
Well, I have opened a ticket https://issues.jenkins.io/browse/JENKINS-68265 about this, but I have been told that the problem might be misconfiguration

Where communities thrive

People

Repo info

Activity