Automatically replace AWS EC2 instances in AutoScaling groups with cheapest compatible spot instances https://autospotting.org
i have changed the code to directly wait for spot instance ready and use it to replace ondemand,
run time is 40 to 50 secs
tomorrow i will try to launch multiple instances for the same asg
currently i have put all code inside
run time is 40 to 50 secs
tomorrow i will try to launch multiple instances for the same asg
currently i have put all code inside
handleNewOnDemandInstanceLaunch but should be better to create a method ad hoc: log.Printf("%s instance %s belongs to an enabled ASG and should be "+
"replaced with spot, attempting to launch spot replacement",
i.region.name, *i.InstanceId)
if spotInstanceID, err := i.launchSpotReplacement(); err != nil {
log.Printf("%s Couldn't launch spot replacement for %s",
i.region.name, *i.InstanceId)
return err
} else {
log.Printf("Waiting for spot instance %s to be in status running", *spotInstanceID)
err := r.services.ec2.WaitUntilInstanceRunning(
&ec2.DescribeInstancesInput{
InstanceIds: []*string{spotInstanceID},
})
if err != nil {
log.Printf("Issue while waiting for spot instance %v to start: %v",
spotInstanceID, err.Error())
return err
}
if err := r.scanInstance(spotInstanceID); err != nil {
log.Printf("%s Couldn't scan instance %s: %s", i.region.name,
*spotInstanceID, err.Error())
return err
}
spotInstance := r.instances.get(*spotInstanceID)
if _, err := spotInstance.swapWithGroupMember(i.asg); err != nil {
log.Printf("%s, couldn't perform spot replacement of %s ",
i.region.name, *i.InstanceId)
return err
}
this way spot "life" outside ASG should be nearly zero, but as downside we will take more time to replace multiple instance belonging to the same asg
hi @cristim i noticed a strange thing:
at 08:22 i had an Instance Rebalance Recomendation, so AS removed it from the ASG, but at 09:35 instance is still running. I think that AWS IRR are not 100% accurate, instance may be terminated or not....
the problem is that we remove tag
at 08:22 i had an Instance Rebalance Recomendation, so AS removed it from the ASG, but at 09:35 instance is still running. I think that AWS IRR are not 100% accurate, instance may be terminated or not....
the problem is that we remove tag
launched-for-asg as for spot, but this way if they are not terminated we end with unused spot instances that will never be terminated by AS (they are like "ghost" instances)
probably best approach should be, do not remove tag
launched-for-asg if event is IRR, what do you think ?
@/all Tomorrow I'll resume the monthly AutoSpotting community calls.
Feel free to join our next session tomorrow at 13:00 UTC if you'd like to get in touch and discuss about AutoSpotting.
Agenda:
- current state of the project and ongoing work
- near-term plans
- free discussion about the project
Looking forward to seeing you.
Best regards,
Cristian
Zoom Meeting URL: https://zoom.us/j/6950084333?pwd=RFplNUhydSsrTlFKWFJaMkczdkU4QT09
Meeting ID: 695 008 4333
Passcode: 9gu6f
@cristim, i made some real test with the previous posted changes to the code (and some other too), seems to work fine, replaced 10 instances, belonging to the same ASG, without any problem.
It took nearly 8 min to replace them all (launch spot + replace).
Only AS run that can work in parallel for the same ASG are the one that send message to the SQS Queue or that terminate instances.
All others are triggered by SQS and work in a sequential way (for the same ASG and Region).
It took nearly 8 min to replace them all (launch spot + replace).
Only AS run that can work in parallel for the same ASG are the one that send message to the SQS Queue or that terminate instances.
All others are triggered by SQS and work in a sequential way (for the same ASG and Region).
i can create a PR so you can have a look at the code, let me know..
(i have changed the logic in
waitForInstanceStatus and gained 10sec for run)
Have you rebase it to my other draft PR?
i need to do this in order to avoid processing spot launch event that do not came from sqs
when in ondemand event we launch the spot, then wait for it and do the replacement, but spot launch trigger another AS run
But I'll try it out tomorrow
we should write some workflows for every trigger events to better understood it
tomorrow i am on holiday, i will try to create them
new workflows
(i have omitted lifecycle and termination events for simplicity)
(i have omitted lifecycle and termination events for simplicity)
@cristim i have create the PR:
AutoSpotting/AutoSpotting#462
it's based on your
AutoSpotting/AutoSpotting#462
it's based on your
sqs-workflow-improvements branch, this way is easier to find the changes
it's a bit slower than I hoped/expected but it's a much smoother ride than what we had before and the Spot instances were all attached immediately to the ASG
I'd probably split it by component to make it easier for them to review
example:
the change for adding ASG LifeCycleHook (create, describe, delete) involve
the change for adding ASG LifeCycleHook (create, describe, delete) involve
moto/autoscaling/models.py , and the same file is involved for the fix to attach_instances
and looking at moto current open PR (91) i fear that PR merging will be slow....
1 reply