Your understanding regarding Max Idle Minutes Before Scaledown is correct. The plugin will terminate the node only when it has been idle for more than Max Idle Minutes Before Scaledown while considering minimum and maximum cluster size. Is this a recurring issue? Can you share the steps to replicate along with the plugin configurations you have in place so we can dive deeper into this?

Just adding more details here to highlight the point of contention :

1. Build is executing on the EC2 instance
2. The user tries to delete their instance by deleting the agent from Jenkins console
3. The instance gets removed from Jenkins

Current Scenario: 4. Since the node was intentionally removed from Jenkins. The target capacity is unchanged and user can re-schedule the build if they wish to.

Now with your change: 4. Build gets re-scheduled and Jenkins immediately provisions another EC2 instance and increases the target capacity so that it could execute the re-scheduled build.

I will spend some more time to see why ChannelTermination is not raised in some scenarios to see if we can fix that and talk to few more customers to gather their inputs. Feel free to let me know your thoughts around this.

@SrodriguezO Thanks for the PR and your response. By manually deleting I meant, changing plugin config for scaling down or deleting agent. IIRC this would result in just having SimpleOfflineCause.

Any reason why we didn't include SimpleOfflineCause but removed the condition forChannelTermination? Are there any unintentional exceptions that we will be including as a result of this?

I'm okay with adding this however I will have to run play around with this more and see if there are any side affects to it. Meanwhile can you update the PR with the integ tests AutoResubmitIntegrationTest.

When a node gets interrupted unintentionally, it raises OfflineCause.ChannelTermination exception. The job is re-queued using the EC2FleetAutoResubmitComputerLauncher as we resubmit the job when we encounter OfflineCause.ChannelTermination exception. However, this does not remove the EC2 instance from jenkins nodes. EC2FleetCloud nanny then sees that instance is missing in EC2 and removes it from the jenkins nodes. We then encounter hudson.slaves.OfflineCause$SimpleOfflineCause exception.

If you see hudson.slaves.OfflineCause$SimpleOfflineCause exception happen when instance is abruptly terminated, you should ideally see OfflineCause.ChannelTermination exception in above logs. If this is incorrect, please let me know the steps to reproduce it. I have tried multiple ways and this was always the case.

The reason we didn't include hudson.slaves.OfflineCause$SimpleOfflineCause was that the users most often terminate the instance (manually or using plugin configs) to stop the build and they don't want the build to be resubmitted. hudson.slaves.OfflineCause$SimpleOfflineCause is a result of user intentionally terminating the node unlike OfflineCause.ChannelTermination which happens when unexpected interruptions happen.

I would like to know if you have any steps to reproduce this. This would be helpful.

Issue #, if available: Automate creating binaries for multiple platforms

Description of changes: Create make target build-binaries to help create binaries for multiple platform

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Automate creating binaries for multiple platforms. Create make target build-binaries to help create binaries for multiple platform

Automate creating binaries for multiple platforms. Create make target build-binaries to help create binaries for multiple platform

Issue #, if available: Automate creating binaries for multiple platforms

Description of changes: Create make target build-binaries to help create binaries for multiple platform

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

It looks like build parameters are lost when a job is resubmitted due to SpotFleet killing the host it was running on.

It's kind of uninteresting but, just in case, here is the log: https://gist.github.com/Liath/bba5290f1bf0665ee61d6ccd1fc3020e

The Jenkinsfile of the job is something like this:

pipeline {
  agent {
    label "ec2-fleet"
  }

  parameters {
    string(name: 'template', defaultValue: 'false', description: 'a path')
    string(name: 'stage', defaultValue: 'dev', description: 'environment')
  }

  stages {
    stage('Build') {
      when { expression { params.template != 'false' } }

      steps {
        withAWS(region: 'us-west-2', credentials: 'aws') {
          sh "long-running --stage ${params.stage}' ${params.template}"
        }
      }
    }
  }
}

I'm closing this for now. Please let us know if this is still the issue and reopen it.

Issue #, if available: N/A

Description of changes:

• Remove .DS_store files
• Fix Travis on PR runs
• Upgrade go to 1.15

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The issue related to class instantiation for TrileadSupport was resolved in ssh-slave 1.23.

More details here: https://issues.jenkins-ci.org/browse/JENKINS-44893

This looks like it's just classes not having been loaded yet, so is just unnecessary noise and happens whenever the instances get terminated/interrupted. There seems to be loading issue with Trilead.

Aug 17, 2020 1:35:30 PM hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager getTrileadSupport
WARNING: Could not create Trilead support class. Using legacy Trilead features
java.lang.ClassNotFoundException: hudson.plugins.sshslaves.verifiers.JenkinsTrilead9VersionSupport
        at org.jenkinsci.maven.plugins.hpi.JettyAndServletApiOnlyClassLoader.findClass(JettyAndServletApiOnlyClassLoader.java:38)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager.createVersion9Instance(TrileadVersionSupportManager.java:52)
        at hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager.getTrileadSupport(TrileadVersionSupportManager.java:32)
        at hudson.plugins.sshslaves.verifiers.SshHostKeyVerificationStrategy.getPreferredKeyAlgorithms(SshHostKeyVerificationStrategy.java:68)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:796)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

The issue related to class instantiation for TrileadSupport was resolved in ssh-slave 1.23.

More details here: https://issues.jenkins-ci.org/browse/JENKINS-44893

Integration tests were failing intermittently as nodes were terminated by Jenkins due to timeout. Fixing this by breaking sleep into multiple parts

Integration tests were failing intermittently as nodes were terminated by Jenkins due to timeout. Fixing this by breaking sleep into multiple parts

Been having this problem of Jenkins master not able to connect to replaced spot instances or new instances added during scale up.

Whenever a spot instance is terminated & replaced, jobs fail to run on the new instance slave spot instance with this error:

hudson.remoting.Channel$CallSiteStackTrace: Remote call to i-09e3xxxxxxxxxxxx Caused: java.io.IOException: Remote call on i-09e3xxxxxx failed

Would appreciate any help in resolving this

I'm resolving this for now. Please feel free to reopen when you have additional details.

Excess capacity isn't immediately pruned when max size is decreased or a fleet greater than max allowed was attached to the plugin.

Expected Behavior:

When max is decreased in the fleet plugin settings, the plugin will prune the nodes which are idle. If the nodes are currently busy, they will NOT be pruned and the build would continue. However, once the build is complete and nodes are idle - they will be removed.

To reproduce:

1. Create new spot request maually for use in Jenkins
2. Add fleet to Jenkins via this plugin, scaling from 1-4
3. (Fleet is added. At some point it scales up to 4 nodes)
4. Remove fleet; add "new" fleet to Jenkins -- same one -- but now scale from 1-2
5. (Fleet is added. Remains at 4 nodes indefinitely)
6. Reboot Jenkins just in case
7. (Remains at 4 nodes)

Expected behavior:

• If newly-added fleet is greater in size than the max, automatically scale it down to the max size.
• i.e. Fleet size immediately becomes 2 nodes

Actual behavior:

• Fleet remains at the previous max size -- 4 in this case -- even though queue has been empty for 6x the scale-down timeout

Versions

• Jenkins 2.190.2
• EC2 Fleet Jenkins Plugin 1.16.1

We need to have only unit tests but no integration tests to test the IdleRententionStrategy. It is important to test idle nodes are terminated as expected.

We need to have only unit tests but no integration tests to test the IdleRententionStrategy. It is important to test idle nodes are terminated as expected.

Excess capacity isn't immediately pruned when max size is decreased or a fleet greater than max allowed was attached to the plugin.

Expected Behavior:

When max is decreased in the fleet plugin settings, the plugin will prune the nodes which are idle. If the nodes are currently busy, they will NOT be pruned and the build would continue. However, once the build is complete and nodes are idle - they will be removed.

When clicking on Configure for a Fleet Cloud node, the following error is thrown:

No page found 'configure-entries.jelly' for class com.amazon.jenkins.ec2fleet.EC2FleetNode

An argument can be made that the user shouldn't be configuring individual nodes, but the node-specific settings (e.g. Tool Locations) do not appear under the Spot Fleet Configuration screen.

I'm closing the issue as we have not heard back from you. Please feel free to reopen the issue by adding more details.

This looks like it's just classes not having been loaded yet, so is just unnecessary noise and happens whenever the instances get terminated/interrupted. There seems to be loading issue with Trilead.

Aug 17, 2020 1:35:30 PM hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager getTrileadSupport
WARNING: Could not create Trilead support class. Using legacy Trilead features
java.lang.ClassNotFoundException: hudson.plugins.sshslaves.verifiers.JenkinsTrilead9VersionSupport
        at org.jenkinsci.maven.plugins.hpi.JettyAndServletApiOnlyClassLoader.findClass(JettyAndServletApiOnlyClassLoader.java:38)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager.createVersion9Instance(TrileadVersionSupportManager.java:52)
        at hudson.plugins.sshslaves.verifiers.TrileadVersionSupportManager.getTrileadSupport(TrileadVersionSupportManager.java:32)
        at hudson.plugins.sshslaves.verifiers.SshHostKeyVerificationStrategy.getPreferredKeyAlgorithms(SshHostKeyVerificationStrategy.java:68)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:796)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

@VackarAfzal . In my case Target capacity and min size works weirdly or please correct me if my understanding is wrong . When I keep the Target capacity in jenkins as 2 and min size as 0 . It immediately sets the target capacity to two in the spot fleet when there is a job which does not require that much . is it supposed to create just one and scale up based on demand ???

Originally posted by @ebalakumar in https://github.com/jenkinsci/ec2-fleet-plugin/issues/52#issuecomment-482520557

I am assuming your question was answered. If not, please let us know.

@Liath I tried to replicate this bug. Steps:

1. Started a parametrized build on instance
2. The build starts executing
3. Terminate the instance abruptly
4. Build gets resubmitted
5. New instance picks up the build.

However, I do see the parameters were also passed and not lost. I would need more details here for reproducing this bug.

Adding imuqtadir as maintainer for project

Adding imuqtadir as maintainer for project

<!-- This PR template only applies to permission changes. Ignore it for changes to the tool updating permissions in Artifactory -->

Description

Please add permissions to @imuqtadir for https://github.com/jenkinsci/ec2-fleet-plugin Existing maintainers: @terma <!-- fill in description here, this will at least be a link to a GitHub repository, and often also links to hosting request, and @mentioning other committers/maintainers as per the checklist below -->

Submitter checklist for adding or changing permissions

<!-- Make sure to implement all relevant entries (see section headers to when they apply) and mark them as checked (by replacing the space between brackets with an "x"). Remove sections that don't apply, e.g. the second and third when adding a new uploader to an existing permissions file. -->

Always

• [x] Add link to plugin/component Git repository in description above

When adding new uploaders (this includes newly created permissions files)

• [x] Make sure to @mention an existing maintainer to confirm the permissions request, if applicable
• [x] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
• [x] Make sure to @mention the users being added so their GitHub account names are known if they require GitHub merge access (see below).
• [x] All newly added users have logged in to Artifactory at least once

Reviewer checklist (not for requesters!)

• [ ] Check this if newly added person also needs to be given merge permission to the GitHub repo (please @ the people/person with their GitHub username in this issue as well). If needed, it can be done using an IRC Bot command
• [ ] Check that the $pluginId Developers team has Admin permissions while granting the access.
• [ ] In the case of plugin adoption, ensure that the Jenkins Jira default assignee is either removed or changed to the new maintainer.
• [ ] If security contacts are changed (this includes add/remove), ping the security officer (currently @daniel-beck) in this pull request. If an email contact is changed, wait for approval from the security officer.

There are IRC Bot commands for it

<!-- This PR template only applies to permission changes. Ignore it for changes to the tool updating permissions in Artifactory -->

Description

Please add permissions to @imuqtadir for https://github.com/jenkinsci/ec2-fleet-plugin Existing maintainers: @terma <!-- fill in description here, this will at least be a link to a GitHub repository, and often also links to hosting request, and @mentioning other committers/maintainers as per the checklist below -->

Submitter checklist for adding or changing permissions

<!-- Make sure to implement all relevant entries (see section headers to when they apply) and mark them as checked (by replacing the space between brackets with an "x"). Remove sections that don't apply, e.g. the second and third when adding a new uploader to an existing permissions file. -->

Always

• [x] Add link to plugin/component Git repository in description above

When adding new uploaders (this includes newly created permissions files)

• [x] Make sure to @mention an existing maintainer to confirm the permissions request, if applicable
• [x] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
• [x] Make sure to @mention the users being added so their GitHub account names are known if they require GitHub merge access (see below).
• [x] All newly added users have logged in to Artifactory at least once

Reviewer checklist (not for requesters!)

• [ ] Check this if newly added person also needs to be given merge permission to the GitHub repo (please @ the people/person with their GitHub username in this issue as well). If needed, it can be done using an IRC Bot command
• [ ] Check that the $pluginId Developers team has Admin permissions while granting the access.
• [ ] In the case of plugin adoption, ensure that the Jenkins Jira default assignee is either removed or changed to the new maintainer.
• [ ] If security contacts are changed (this includes add/remove), ping the security officer (currently @daniel-beck) in this pull request. If an email contact is changed, wait for approval from the security officer.

There are IRC Bot commands for it

By clicking 'Test Connections' button, users can verify if they are missing any AWS policies required by the plugin.

I had previously submitted a documentation patch related to proper AWS permissions: https://github.com/jenkinsci/ec2-fleet-plugin/pull/56/files

This documentation appears to have been removed from the README.

I think it would be better to provide a check for proper permissions as part of the "Test Connection" button in the plugin configuration. I assume this would be done best by querying the permissions of the user, not by something like attempting to "modify" the spot request to its same values. This way, you have a test not only that the fleet can be used but that the entire plugin will function as expected.

By clicking 'Test Connections' button, users can verify if they are missing any AWS policies required by the plugin.

Hello.

Not sure, but it looks like in this particular case is slightly different from https://github.com/jenkinsci/ec2-fleet-plugin/issues/121 .

Basically, one of the things which happens from time to time is following:

1. Node is provisioned
2. Node gets job
3. There happens I/O error
4. build fails and getting resubmitted
5. Connection restores

So, c.a.j.e.EC2FleetAutoResubmitComputerLauncher#afterDisconnect does resubmit - which is actually good, but it does not count that node is back there in some time.

So what do you think? Should it rather wait for node to come back online, with limited amount of retries, etc or rather current behaviour is proper?

Jenkins ver. 2.191 Plugin ver. 1.10.2

./jenkins.log:2019-09-12 05:32:20.972+0000 [id=5104742]	INFO	h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel i-0978b0726c81e4fda
./jenkins.log:2019-09-12 05:32:20.978+0000 [id=5104742]	INFO	c.a.j.e.EC2FleetAutoResubmitComputerLauncher#afterDisconnect: Unexpected FleetCloud i-0978b0726c81e4fda termination,  resubmit
./jenkins.log:2019-09-12 05:32:20.983+0000 [id=5104742]	INFO	c.a.j.e.EC2FleetAutoResubmitComputerLauncher#afterDisconnect: Unexpected FleetCloud i-0978b0726c81e4fda termination, resubmit org.jenkinsci.plugins.workflow.job.WorkflowJob@2dd7fc62[%%%] with actions []
./jenkins.log:2019-09-12 05:32:20.988+0000 [id=5104742]	INFO	c.a.j.e.EC2FleetAutoResubmitComputerLauncher#afterDisconnect: Unexpected FleetCloud i-0978b0726c81e4fda termination, resubmit finished
./jenkins.log:2019-09-12 05:33:05.427+0000 [id=54]	INFO	c.a.j.e.IdleRetentionStrategy#check: Reconnecting to instance: FleetCloud i-0978b0726c81e4fda
./jenkins.log:2019-09-12 05:33:05.427+0000 [id=54]	INFO	hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect i-0978b0726c81e4fda
./jenkins.log:[09/12/19 05:33:06] SSH Launch of i-0978b0726c81e4fda on %%% failed in 646 ms
./jenkins.log:2019-09-12 05:33:17.135+0000 [id=48]	INFO	c.a.j.e.IdleRetentionStrategy#check: Reconnecting to instance: FleetCloud i-0978b0726c81e4fda
./jenkins.log:2019-09-12 05:33:17.135+0000 [id=48]	INFO	hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect i-0978b0726c81e4fda
./jenkins.log:[09/12/19 05:33:17] SSH Launch of i-0978b0726c81e4fda on %%% failed in 409 ms
./jenkins.log:2019-09-12 05:35:17.110+0000 [id=51]	INFO	c.a.j.e.IdleRetentionStrategy#check: Reconnecting to instance: FleetCloud i-0978b0726c81e4fda
./jenkins.log:2019-09-12 05:35:17.110+0000 [id=51]	INFO	hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect i-0978b0726c81e4fda
./jenkins.log:[09/12/19 05:35:20] SSH Launch of i-0978b0726c81e4fda on %%% completed in 3,336 ms

I'm hoping the issue is resolved. Please feel free to reopen the issue with the suggested workaround doesn't work well with you.

Yes you are right. EC2 Fleet plugin supports ASG and Spot Fleet and brings all the benefits associated to it whereas EC2 Plugin uses RunInstance EC2 API which requires ad-hoc instances. In EC2 Fleet, the desired capacity of the fleet is calculated by the plugin based on the builds and other settings provided by the users and the fleet target capacity is adjusted based on it. I will keep this issue open to update README with the comparison of both the plugins so that it's helpful to others too.

Add Amazon code of conduct

Add Amazon code of conduct

Added contributing instructions so it's easier for future contributors.

Publish CONTRIBUTING.md for helping people contribute effectively and easily.