Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    tburow
    @tburow
    ssh -o StrictHostKeyChecking=yes -i /tmp/ec2_1432213094080254131.pem ubuntu@10.86.5.96 -p 22 java -jar /tmp/remoting.jar -workDir /tmp
    No ECDSA host key is known for 10.86.5.96 and you have requested strict checking
    Alex Earl
    @slide
    Are you using jenkins/ssh-agent?
    tburow
    @tburow
    yes
    tburow
    @tburow
    just for the record - this plugin is a love/hate - love the plugin - hate when it gets updated because its always breaking on updates...
    Alex Schittko
    @alex4108
    Yeesh and here I was thinking I did something bad updating plugins
    Narayanan Singaram
    @narayanan
    In our environment we use ec2-plugin to create windows slaves and execute builds on that node. When launching windows slaves, we noticed the initial WinRM process opens 800+ connection to WinRM port (5985 / 5986) for each slave node.. Upon looking into the code, I see Apache DefaultHttpClient is being used, which does not use any HTTP connection pooling? Any specific reason to not using HTTP connection pooling ? Does WinRM not support re-using connections? This is becoming a serious issue in our environment, if we attempt to launch 18 or more nodes, with in couple of minutes of launching, most of the nodes goes offline due to connection termination error... after analyzing the network transmission data using tcpdump, we found that due to too many short lived connections to WinRM, number of available ports is getting exhausted and ports in TIME_WAIT state getting re-used in Jenkins server, Windows just simply does not acknowledge requests when it detects a port re-use.. Has this issue been observed earlier? Is it worth replacing the DefaultHttpClient with a custom client with pooling http connection manager?
    8 replies
    Raihaan Shouhell
    @res0nance
    Looks like you have an older ssh client, the downgrade seems to be a known issue as well. The security fixes require some work it seems. These issues are being looked into.
    Sasha Miroshnychenko
    @almiroshnich
    Hi
    Has anybody observed the behavior of the plugin when it's getting super slow (scales literally by 1 slave at a time) on launching EC2s with spot-block enabled when AWS has a shortage with spot availability? Usually, the launching of the slaves happens by big chunks when there is a huge queue of waiting for executor builds...
    5 replies
    donepudi369
    @donepudi369

    @res0nance @slide @alok0310 - I have tried connecting to windows 2016 using the ssh method described above and copying the ssh key but I am still unable to connect. Can you please let me know if there are any other steps to be followed?

    Steps followed :

    1. Created a Windows EC2 machine and run the poweshell script provided by @res0nance and made sure I am able to connect via ssh
    2. Created and AMI and configured it in the Jenkins to use Unix method
    3. Provisioned an agent but Jenkins is unable to connect

    Note: I have tested the ssh connectivity of the new agent created by jenkins and I am able to login but Jenkins is unable to login

    Error:
    INFO: The instance EC2 (AWS-sandbox) - windows-2016 (i-05a178d76ba4XXXX) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: The instance console is blank. Cannot check the key. The connection to EC2 (AWS-sandbox) - windows-2016 (i-05a178d76ba4XXXX) is not allowed
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: Failed to connect via ssh: There was a problem while connecting to XX.XX.XX.XX:22
    Jun 02, 2020 6:20:15 PM hudson.plugins.ec2.EC2Cloud
    INFO: Waiting for SSH to come up. Sleeping 5.
    Jun 02, 2020 6:20:16 PM hudson.plugins.ec2.EC2Cloud
    INFO: Connecting to XX.XX.XX.XX on port 22, with timeout 10000.

    Alex Earl
    @slide
    Did you open port 22 in your security group?
    Also, it looks like you have the stricter key checking method turned on, I am not sure how to implement that method
    pyieh
    @pyieh
    @res0nance I'm wondering if we can get some visibility on our PR here: https://github.com/jenkinsci/ec2-plugin/pull/448/ . If it's all good can we get it merged?
    bsubbaraman
    @bsubbaraman
    I also had a lot of problems trying to do the described windows ssh method -- assuming all your ports/etc are set up correctly, it worked for me when i changed cloud configuration --> advanced --> Host Key Verification Strategy to accept-enw
    *accept-new (instead of the default 'check-new hard')
    donepudi369
    @donepudi369

    @slide - Yes port 22 is open. I am able to connect to the server using ssh directly.

    I have tried changing the "Host Key Verification Strategy" to 'accept-new' and off but AWS is terminating the instances within few minutes. Maybe it is thinking as a "man-in-the-middle" attack since I am using public IP to connect.

    Are you using public IP or private IP for connection?

    bsubbaraman
    @bsubbaraman
    i am using private IP
    and launching instances into my VPC (with subnet ID also set in cloud config)
    donepudi369
    @donepudi369
    I will give a try with private IP and specifying the subnet ID
    bsubbaraman
    @bsubbaraman
    I have a separate question: whenever I try to launch a second node with a different label (but using the same AMI), it never launches and says 'all nodes of label 'mylabel' are offline. I have no global instance cap or other instance caps set. Has anyone encountered that?
    similar issue is documented here - running the grooving script reference also returns '1' for me (not '0') https://groups.google.com/forum/#!topic/jenkinsci-users/fSKahUyrpqs
    donepudi369
    @donepudi369
    Jenkins is able to connect to the "Windows server" after changing the Host Key Policy to "accept-new".
    Thanks for your help
    donepudi369
    @donepudi369

    I also want to test regular connection to Windows EC2 server using "winrm".

    Does anyone have details on how to get it working? I applied all the settings as described in the plugin and created an image but it always hangs at "connecting to (XX.XX.XX.XX) with WinRM as Administrator" and never connects.

    bsubbaraman
    @bsubbaraman
    ^ i got stuck at the same spot. spent forever on it, had all ports open, could connect to the relevent ports using telnet, but always hangs. That's why i switched to the ssh method
    Glad to hear you got connected!
    Alex Earl
    @slide
    I haven't gotten WinRM to work correctly
    eduardoalmeida
    @eduardoalmeida

    I have the same problem as @donepudi369 and @bsubbaraman. I tried to figured it out and I discovered that it is hanging when creating the SMBClient.
    I added some logs to the code:
    log.log(Level.FINE, "Inside WinConnection constructor");
    this.host = host;
    this.username = username;
    this.password = password;
    log.log(Level.FINE, "Creating SMBClient");
    this.smbclient = new SMBClient();
    log.log(Level.FINE, "SMBClient created");

    I can see the Inside WinConnection constructor and Creating SMBClient but it seems like SMBClient() never returns. The "SMBClient created" is never logged.

    donepudi369
    @donepudi369
    @eduardoalmeida - Did you try creating the smb client in the server and creating an AMI from it? so that it doesn't need to create again?
    John LaBarge
    @johnlabarge
    We really need something that doesn't require a PEM file to login into the instance and rather generates the ssh keys on the fly. I've seen some indications that others need this as well and was thinking of doing a PR
    John LaBarge
    @johnlabarge
    Our environment doesn't allow us to use the AWS keys. Instead we use IAM credentials. For ssh purposes we could use the user data to add the generated key for a user and login to the instance from the agent that way.
    tburow
    @tburow
    did the "StrictHostKeyChecking" options ever get fixed? last time I attempted to run the updated plugin it failed miserably because the ssh options being presented are not compliant with openssh
    John LaBarge
    @johnlabarge
    I haven't seen that error. My problem is it looks for the ssh key using the AWS api and doesn't find it. But it doesn't need it to create the instance anyway.
    dancsa
    @dancsa

    Hi, i was tasked to get ec2 in windows working. After spending some hours trying winRM, switched to the openssh was. (i havent read this chat until now, btw on recent enough windows (win10 or win serv 2019) the SSH can be installed with more easily than said above)
    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
    Start-Service sshd
    Set-Service -Name sshd -StartupType 'Automatic'
    New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

    It hasn't appeared in my mind to run the plugin in linux mode so i just started a java IDE, and rework the EC2WindowsLauncher.java to use SSH (copypasta driven development from Ec2UnixLauncher.java)
    It is on https://github.com/ultinous-dancsa/ec2-plugin/tree/windows-over-ssh-poc
    (its code quality is somewhere between the spagetti and "i just want a working PoC within 3 hours with no prior knowledge")
    It works with password auth, the ami only had the ssh, and java install and a user creation

    Do you plan to implement something like this in the plugin, or i've just read above the unix setting works for windows too if ssh is installed?

    Alex Earl
    @slide
    I'm sure a PR would be welcome
    dancsa
    @dancsa
    I don't really have much freetime for coding, but if nobody takes it over, maybe I'll try to hammer it further.
    Sickafant
    @Sickafant
    Hello. Is anyone aware if there's been interest in supporting AMI aliases? https://aws.amazon.com/about-aws/whats-new/2020/05/amazon-ec2-now-supports-aliases-for-amis/
    pyieh
    @pyieh
    Can I get some visibility on my PR here https://github.com/jenkinsci/ec2-plugin/pull/448/ ? It's been approved and I'd like to get it merged. It fixes a bug we've seen where orphan nodes aren't reconnected if the total number of existing nodes has hit the instance cap.
    mikelmao
    @mikelmao
    Iv been stuck for 3 days trying to get EC2 plugin to work with windows instance
    As suggest by @res0nance I have ran the script to install SSH and try it that way, i am able to connect to SSH from my local computer but i am getting authentication failed if EC2 plugin tries to connect
    From local machine i am connecting with plain username and password, i think maybe im doing something wrong with the whole SSH key? I have no experience with SSH keys so im sure im doing something wrong
    I copied the .pem contents and pasted where it says "public-key-here" in the script, is there anything else i need to do?
    image.png
    mikelmao
    @mikelmao
    Actually i see now that the file i got from EC2 is a private key
    mikelmao
    @mikelmao
    Iv made a public key from it but still same issue :(
    mikelmao
    @mikelmao
    Seems like i got a step further, though now getting this error
    image.png
    mikelmao
    @mikelmao
    Seems like its working now after turning fingerprint checking off :)
    bunchopunch
    @bunchopunch
    Hey, y'all. We're still on 1.50.2 and this morning our EC2 plugin based instances seem to have suddenly stopped allowing us to validate their key during the spin up. They come back with this error:
    The instance EC2 Amazon-Linux 2 (i-xxxxxxxxxxxxx) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
    ...
    INFO: The instance console is blank. Cannot check the key. The connection to EC2 Amazon-Linux 2 (i-xxxxxxxxxxxxx) is not allowed
    
    ...
    HTTP ERROR 404 Not Found
    What's the best place to start in terms of understanding what could be causing this?
    Alex Earl
    @slide
    It looks like you may need to change your key checking in the cloud config
    bunchopunch
    @bunchopunch
    That was sort of what I was thinking as well. I'm just waiting until after hours to make any configuration changes now.