I'd start by looking at the Github OAuth plugin, seeing what version you have and if you can locate any relationship between it and the problem.
@/all JNLP1..3 protocols were removed in 3.40, and they will be removed from the next Jenkins weekly: https://groups.google.com/forum/#!topic/jenkinsci-dev/VBeEhaE7VcQ
hello guys, i want to work on the following newbie-friendly issue, can some guide?
https://issues.jenkins-ci.org/browse/JENKINS-49905
https://issues.jenkins-ci.org/browse/JENKINS-49905
4 replies
Is there a place I can start from? I am studying distributed systems this semester so I have some knowledge about remoting.
@zachary2940 hi, thanks for the interest! The most simple way to start is to try out Jenkins with distributed masters. There is a newly introduced Websocket support, and you can explore it: https://jenkins.io/blog/2020/02/02/web-socket/ . Adding native support to plugins like Swarm Plugin could be a good first step.
Also, there are some open newbie-friendly issues: https://issues.jenkins-ci.org/browse/JENKINS-40700?jql=labels%20%3D%20newbie-friendly%20and%20status%20in%20(Open%2C%20%22To%20Do%22%2C%20Reopened)%20and%20component%20%20%3D%20remoting
Is anybody working on this now https://issues.jenkins-ci.org/browse/JENKINS-40700? Sorry I have no idea how to add a comment directly to the issue
I'm not sure if this has been ever reported or is a known issue but calling
/disconnect or having a computer's availability setting disconnect an agent consistently causes a ClosedChannelException so in essence SlaveComputer.disconnect will cause this exception to happen. This seems to only happen to "JNLP" connections AFAICT. Stacktrace below:java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:784)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
at hudson.remoting.Channel.close(Channel.java:1452)
at hudson.remoting.Channel.close(Channel.java:1405)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:844)
at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:105)
at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:753)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Hi all. I'm trying to connect a WINDOWS 10 as a agent by Java web Start method to a Linux Master. But I'v got a response "provided port:6200 is not reachable". I'v closed all firewall programs and I'm sure that port 6200 is not occupied.Is there anyone encoutered issue like this: 三月 02, 2020 11:12:34 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
信息: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
三月 02, 2020 11:12:36 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
警告: Connection refused: connect
三月 02, 2020 11:12:36 下午 hudson.remoting.jnlp.GuiListener$2 run
严重: http://123.126.106.85:8080/ provided port:6200 is not reachable
java.io.IOException: http://123.126.106.85:8080/ provided port:6200 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314)
at hudson.remoting.Engine.innerRun(Engine.java:690)
at hudson.remoting.Engine.run(Engine.java:518)
信息: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
三月 02, 2020 11:12:36 下午 org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
警告: Connection refused: connect
三月 02, 2020 11:12:36 下午 hudson.remoting.jnlp.GuiListener$2 run
严重: http://123.126.106.85:8080/ provided port:6200 is not reachable
java.io.IOException: http://123.126.106.85:8080/ provided port:6200 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314)
at hudson.remoting.Engine.innerRun(Engine.java:690)
at hudson.remoting.Engine.run(Engine.java:518)
It's probably a network configuration issue, but it's hard to suggest what the issue might be. I recommend you follow some network configuration and reachability troubleshooting. Use some other tools to try to connect to that port. Try verifying traffic between the computers. Try using a different port. It looks like you're hitting the very basic, simple port connection test and it's failing.
Hi! I have open this ticket today about an issue with remoting 4.2.1 and and 4.3 (seems a regression from 4.2) related to websocket : https://issues.jenkins-ci.org/browse/JENKINS-62576
I'm aware that the WS feature is still in beta, but it literally saved me to avoid networking issue (firewall, K8S ingress etc...) :)
Except the previous issue with large payload I didn't had previously any issue with connection or stability. Now I'm the connection is crashing in the middle of the build. I had sadly to rollback to 2.222.3
I've put all the details on the ticket, I can help debug or provide more logs if needed.
Thanks!
Hi! I have nginx proxy setup, using normal JNPL agents. Now I switched (trying) to WebSocket, but receiving a strange error
JNLP file https://xzy.com/jenkins/computer/vsb-10/slave-agent.jnlp?encrypt=true has invalid arguments: [89aacf49720cd4550ead6eb23060e995201842915acddff53776a6baa69177bd, node-10, -webSocket, -workDir, c:\workdir\vsb\jenkins, -internalDir, remoting, -url, https://xyz.com/jenkins/, -url, http://xzy-1.com:8080/jenkins/, -headless, -workDir, c:\workdir\vsb\jenkins, -internalDir, remoting]
Most likely a configuration error in the master
-webSocket supports only a single -urlAny suggestions I should try?
Thanks!
3 replies
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
INFO: Agent discovery successful
Agent address: jenkins.operations.svc.cluster.local
Agent port: 50000
INFO: Handshaking
INFO: Connecting to jenkins.operations.svc.cluster.local:50000
INFO: Trying protocol: JNLP4-connect
Apr 16, 2019 1:29:01 PM org.jenkinsci.remoting.protocol.impl.AckFilterLayer abort
WARNING: [JNLP4-connect connection to jenkins.operations.svc.cluster.local/10.19.251.169:50000] Incorrect acknowledgement sequence, expected 0x000341434b got 0x485454502f
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)....
INFO: Agent discovery successful
Agent address: jenkins.operations.svc.cluster.local
Agent port: 50000
INFO: Handshaking
INFO: Connecting to jenkins.operations.svc.cluster.local:50000
INFO: Trying protocol: JNLP4-connect
Apr 16, 2019 1:29:01 PM org.jenkinsci.remoting.protocol.impl.AckFilterLayer abort
WARNING: [JNLP4-connect connection to jenkins.operations.svc.cluster.local/10.19.251.169:50000] Incorrect acknowledgement sequence, expected 0x000341434b got 0x485454502f
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)....
also i am looking to implement dynamic agent provisioning
something in similar terms like kubernetes or ec2
given these plugins are quite difficult to understand
i want to understand easy starting point
Hello,
I'm running jenkins 2.269 in a docker swarm cluster behind traefik 2 and i'm trying to configure the docker plugin 1.2.1 to autoprovision jnlp agents using the inbound agent docker image https://hub.docker.com/r/jenkins/inbound-agent/
the issue that i'm facing is that tcp connection over port 50000 using traefik is not working see https://issues.jenkins.io/browse/JENKINS-62738 so i had to configure my agents to use the websocket. This works fine with permanent agents. But with the docker plugin i keep getting the following error. any thoughts ?
I'm running jenkins 2.269 in a docker swarm cluster behind traefik 2 and i'm trying to configure the docker plugin 1.2.1 to autoprovision jnlp agents using the inbound agent docker image https://hub.docker.com/r/jenkins/inbound-agent/
the issue that i'm facing is that tcp connection over port 50000 using traefik is not working see https://issues.jenkins.io/browse/JENKINS-62738 so i had to configure my agents to use the websocket. This works fine with permanent agents. But with the docker plugin i keep getting the following error. any thoughts ?
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
I don't have any good troubleshooting tips. You'll have to do some deep troubleshooting on this configuration. Something isn't working when it tries to complete the exception. It kind of looks like you haven't gotten it configured correctly to use Websockets. The ConnectionRefusalException is related to the (confusingly misnamed) JNLP Remoting protocols. I don't think it's used in the Websockets implementation, though I'm not sure. If you do figure something out, please share so others can build on that.
Thxs @jeffret-b i'm now able to provision on demand ssh, jnlp and docker in docker agents.
The first issue i had to debug was the use of Websocket in the docker plugin template plugin configuration vs adding and env directive in the dockerfile JENKINS_WEB_SOCKET=true.
this lead me to a better stack trace in jenkins system logs. by this time containers were started but then crash immediately. and that's when i realized that i hit this issue while trying to persist workspaces using volumes jenkinsci/docker-plugin#821 installing the beta-release fixed this and i'm now all set.
The first issue i had to debug was the use of Websocket in the docker plugin template plugin configuration vs adding and env directive in the dockerfile JENKINS_WEB_SOCKET=true.
this lead me to a better stack trace in jenkins system logs. by this time containers were started but then crash immediately. and that's when i realized that i hit this issue while trying to persist workspaces using volumes jenkinsci/docker-plugin#821 installing the beta-release fixed this and i'm now all set.
Hi, Mr. Nenashev.
I'm writing a proposal to expose the agent metrics on the "agent" node and I have some questions.
I think we need to think carefully about the way to deliver and execute the programs because the process or thread to collect and expose the agent metrics must be running even when the connection to the controller is lost, but at the same time, we cannot lose control of the process or thread.
Now, I come up with two possible types of the way to execute the programs. One is simply executing them as a thread or subprocess and the other is creating a UNIX/Windows service. The advantage of launching as a service is that we can keep collecting the metrics even if the agent is restarted while the connection to the controller is lost. But I think launching as a thread is better because launching as a service has a critical problem. Thinking about uninstalling this plugin while the connection is lost, we completely lose control of the service while the service is still running. The same thing can be said for launching as a thread but the thread will stop at least when the agent restarts. It is much better, I think. What do you think about this? And if you know a similar project which handles this problem, I want to know it.
I'm writing a proposal to expose the agent metrics on the "agent" node and I have some questions.
I think we need to think carefully about the way to deliver and execute the programs because the process or thread to collect and expose the agent metrics must be running even when the connection to the controller is lost, but at the same time, we cannot lose control of the process or thread.
Now, I come up with two possible types of the way to execute the programs. One is simply executing them as a thread or subprocess and the other is creating a UNIX/Windows service. The advantage of launching as a service is that we can keep collecting the metrics even if the agent is restarted while the connection to the controller is lost. But I think launching as a thread is better because launching as a service has a critical problem. Thinking about uninstalling this plugin while the connection is lost, we completely lose control of the service while the service is still running. The same thing can be said for launching as a thread but the thread will stop at least when the agent restarts. It is much better, I think. What do you think about this? And if you know a similar project which handles this problem, I want to know it.
I wrote my first proposal draft, and I would like you to review it.
Hi @Aki-7 Looks good, Also possible to just explore and see how you can use these collected (Prometheus) data to create a nice dashboard for Jenkin's admin. Just think of different monitoring we can do using this plugin, and what data needs to fetch using an agent and then create some charts from it would be good idea. Maybe we can also consider it into the proposed schedule.
1 reply
@Aki-7 Hi! If you have some time, it might make sense to take a look at the OpenTelemetry plugin: https://github.com/jenkinsci/opentelemetry-plugin . It also exposes some node/executor metrics in an open format
2 replies
Hello Oleg, participants and other members of channel. Please find some time and review my proposal https://docs.google.com/document/d/1Mq9PAxUbQ_TUu7u_hEGkgzdUQEhXGiCDoYcxISJdda4/edit
1 reply
This year I'm going to fully commit in k8s, Jenkins and implement security web hooks. https://www.jenkins.io/projects/gsoc/2021/project-ideas/jenkins-operator-security/ The main thing which I puzzled at the moment - did i broke down the deliverables correctly?
@/all Jenkins agent user survey! Akihiro Kiuchi, a GSoC 2021 student in Jenkins/CDF, is looking for feedback about Jenkins remoting pain points, and how could we improve agent monitoring and troubleshooting experience with OpenTelemetry. https://docs.google.com/forms/d/e/1FAIpQLSdiuQN3sm2mQ2E86VTXVXu7bf_9C0hVdzhw2_Kvu3DFqL7EZA/viewform