Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Kalby Imanie
    @kalbyimanie
    is it that hard to make it work ?
    Alex Earl
    @slide
    I just found that the WinRM stuff wasn't as stable
    I setup my AMI using packer and installed the OpenSSH-Win32 port and the hostkey into the image. Then you switch the AMI type to Linux in the cloud configuration
    Fabrizio Manfredi Furuholmen
    @thoulen
    @alok0310 that sometime happen when the jenkins queue stuck, do you have also job that are reported completed in the job page and are still running in the left bar ?
    alok0310
    @alok0310
    @thoulen No, I don't think so.
    James Brown
    @Roguelazer
    I just upgraded to 1.50 and Jenkins somehow was no longer able to see the labels on my builders (every job was stuck waiting for a builder to appear with the right label). Downgrading to 1.49.1 fixed it immediately. Anyone seen anything like that?
    Attempting to terminate an instance also crashed with a NPE
    (all on Jenkins LTS)
    Raihaan Shouhell
    @res0nance
    @Roguelazer I have not seen this. Do you have a stacktrace or some way to replicate this?
    tburow
    @tburow
    Im seeing the same behavior with 1.50 - downgraded to fix. could not interact with executors - could not schedule jobs - could not delete executors - could not allocate new executors - got the “oops” window everytime.
    Here is the stack trace
    java.lang.NullPointerException at hudson.plugins.ec2.util.MinimumInstanceChecker.lambda$countQueueItemsForAgentTemplate$8(MinimumInstanceChecker.java:67) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.LongPipeline.reduce(LongPipeline.java:461) at java.util.stream.LongPipeline.sum(LongPipeline.java:419) at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:593) at hudson.plugins.ec2.util.MinimumInstanceChecker.countQueueItemsForAgentTemplate(MinimumInstanceChecker.java:68) at hudson.plugins.ec2.util.MinimumInstanceChecker.lambda$null$11(MinimumInstanceChecker.java:87) at java.util.ArrayList.forEach(ArrayList.java:1257) at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082) at hudson.plugins.ec2.util.MinimumInstanceChecker.lambda$checkForMinimumInstances$12(MinimumInstanceChecker.java:76) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at hudson.plugins.ec2.util.MinimumInstanceChecker.checkForMinimumInstances(MinimumInstanceChecker.java:75) at hudson.plugins.ec2.SlaveTemplate$OnSaveListener.onChange(SlaveTemplate.java:1410) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:81) at jenkins.model.Jenkins.save(Jenkins.java:3385) at jenkins.model.Jenkins.<init>(Jenkins.java:985) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:262)
    looks like MinimumInstanceChecker.java is the problem child
    Raihaan Shouhell
    @res0nance
    @tburow Your large stacktrace should be fixed in 1.50 but there is the other bug which has an open PR we are just waiting for more reviews
    James Brown
    @Roguelazer
    https://gist.github.com/Roguelazer/e81719d2a1d5f0799edbf01f57d145ab is what happens if I try to terminate an instance on 1.50
    I can't get a traceback for no labels matching, but it does recur if I roll forward
    I have every EC2 slave assigned a label and I have a job that uses a label filter to only run on those instances, but it just says "waiting for builder" if I am on 1.50
    ^ @res0nance
    Raihaan Shouhell
    @res0nance
    1.50.1 has been released
    Philip Hyunsu Cho
    @hcho3
    When I upgraded to 1.50 today, I got error "Network interfaces and an instance-level security groups may not be specified on the same request" when my server attempted to launch new EC2 workers. I worked around the problem by downgrading to 1.49.1. So the same configuration worked for 1.49.1 but didn't for 1.50. How should I change my configuration? Any guess what's going on? Ps. Thanks for the great plugin!
    Fabrizio Manfredi Furuholmen
    @thoulen
    can you send you configuration ?
    we aer aslo releaseing 1.50.1 , 1.50 has some heavy bugs
    Philip Hyunsu Cho
    @hcho3
    Should I take the screenshot of the configuration page?
    Fabrizio Manfredi Furuholmen
    @thoulen
    yes it is fine , please open a ticket an attach to it
    Philip Hyunsu Cho
    @hcho3
    Where is the issue tracker? Is it the same as the main Jenkins project?
    Fabrizio Manfredi Furuholmen
    @thoulen
    https://issues.jenkins-ci.org/
    yes it is the same , you have to specify the plugin in the component field
    Philip Hyunsu Cho
    @hcho3
    Thanks. I filed it: https://issues.jenkins-ci.org/browse/JENKINS-62090
    Aditya
    @abaghel
    Guys new here need some help on ec2-plugin groovy script
    after updating to 1.50 getting contractor issues
    parameswaranj
    @parameswaranj
    HI team.. We see the following error with ec2 plugins (ver 1.49.1 and 1.50.1) when trying to launch windows EC2 agents. I raised a JIRA ticket - https://issues.jenkins-ci.org/browse/JENKINS-61006 for the same but haven't heard of any updates on it still. This issue has been noticed only with the windows slaves. Rolling it back to 1.46.1 is bringing the agents up perfectly. Any updates on this would be really appreciated.
    java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:75)
    Caused: com.hierynomus.protocol.transport.TransportException
    at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:78)
    at com.hierynomus.smbj.connection.Connection.send(Connection.java:297)
    at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:305)
    at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:244)
    at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:181)
    Caused: com.hierynomus.smbj.common.SMBRuntimeException
    at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:215)
    at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:112)
    at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:173)
    at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:52)
    at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:292)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    tburow
    @tburow
    1.50.2 release is broken - “accept-new” is not a valid SSH option
    Last stable release is 1.49….
    Fabrizio Manfredi Furuholmen
    @thoulen
    yes for now it is the most safe version
    tburow
    @tburow
    I would recommend recinding 1.50.1 and 1.50.2 versions
    tburow
    @tburow
    tips on manually installing 1.49.1?
    tburow
    @tburow
    Failed Loading plugin Amazon EC2 plugin v1.49.1 (ec2)
    java.lang.IllegalStateException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
    at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
    at hudson.diagnosis.OldDataMonitor.get(OldDataMonitor.java:91)
    at hudson.diagnosis.OldDataMonitor.report(OldDataMonitor.java:223)
    at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:368)
    at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    Caused: com.thoughtworks.xstream.converters.ConversionException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0 : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
    ---- Debugging information ----
    message : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
    cause-exception : java.lang.IllegalStateException
    cause-message : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
    class : hudson.plugins.ec2.PluginImpl
    required-type : hudson.plugins.ec2.PluginImpl
    converter-type : hudson.util.RobustReflectionConverter
    path : /hudson.plugins.ec2.PluginImpl
    line number : 4
    version : not available
    tburow
    @tburow
    ssh -o StrictHostKeyChecking=off -i /tmp/ec2_6874758053699987113.pem ubuntu@10.86.5.96 -p 22 java -jar /tmp/remoting.jar -workDir /tmp
    command-line line 0: unsupported option "off".
    not a single one of the checking options is supported via the Jenkins Docker Container.
    ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_5423098466597994221.pem ubuntu@10.86.5.72 -p 22 java -jar /tmp/remoting.jar -workDir /tmp
    command-line line 0: unsupported option "accept-new
    tburow
    @tburow
    ssh -o StrictHostKeyChecking=yes -i /tmp/ec2_1432213094080254131.pem ubuntu@10.86.5.96 -p 22 java -jar /tmp/remoting.jar -workDir /tmp
    No ECDSA host key is known for 10.86.5.96 and you have requested strict checking
    Alex Earl
    @slide
    Are you using jenkins/ssh-agent?
    tburow
    @tburow
    yes
    tburow
    @tburow
    just for the record - this plugin is a love/hate - love the plugin - hate when it gets updated because its always breaking on updates...
    Alex Schittko
    @alex4108
    Yeesh and here I was thinking I did something bad updating plugins
    Narayanan Singaram
    @narayanan
    In our environment we use ec2-plugin to create windows slaves and execute builds on that node. When launching windows slaves, we noticed the initial WinRM process opens 800+ connection to WinRM port (5985 / 5986) for each slave node.. Upon looking into the code, I see Apache DefaultHttpClient is being used, which does not use any HTTP connection pooling? Any specific reason to not using HTTP connection pooling ? Does WinRM not support re-using connections? This is becoming a serious issue in our environment, if we attempt to launch 18 or more nodes, with in couple of minutes of launching, most of the nodes goes offline due to connection termination error... after analyzing the network transmission data using tcpdump, we found that due to too many short lived connections to WinRM, number of available ports is getting exhausted and ports in TIME_WAIT state getting re-used in Jenkins server, Windows just simply does not acknowledge requests when it detects a port re-use.. Has this issue been observed earlier? Is it worth replacing the DefaultHttpClient with a custom client with pooling http connection manager?
    8 replies
    Raihaan Shouhell
    @res0nance
    Looks like you have an older ssh client, the downgrade seems to be a known issue as well. The security fixes require some work it seems. These issues are being looked into.
    Sasha Miroshnychenko
    @almiroshnich
    Hi
    Has anybody observed the behavior of the plugin when it's getting super slow (scales literally by 1 slave at a time) on launching EC2s with spot-block enabled when AWS has a shortage with spot availability? Usually, the launching of the slaves happens by big chunks when there is a huge queue of waiting for executor builds...
    5 replies
    donepudi369
    @donepudi369

    @res0nance @slide @alok0310 - I have tried connecting to windows 2016 using the ssh method described above and copying the ssh key but I am still unable to connect. Can you please let me know if there are any other steps to be followed?

    Steps followed :

    1. Created a Windows EC2 machine and run the poweshell script provided by @res0nance and made sure I am able to connect via ssh
    2. Created and AMI and configured it in the Jenkins to use Unix method
    3. Provisioned an agent but Jenkins is unable to connect

    Note: I have tested the ssh connectivity of the new agent created by jenkins and I am able to login but Jenkins is unable to login

    Error:
    INFO: The instance EC2 (AWS-sandbox) - windows-2016 (i-05a178d76ba4XXXX) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: The instance console is blank. Cannot check the key. The connection to EC2 (AWS-sandbox) - windows-2016 (i-05a178d76ba4XXXX) is not allowed
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: Failed to connect via ssh: There was a problem while connecting to XX.XX.XX.XX:22
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: Waiting for SSH to come up. Sleeping 5.
    Jun 02, 2020 6:20:16 PM hudson.plugins.ec2.EC2Cloud
    INFO: Connecting to XX.XX.XX.XX on port 22, with timeout 10000.