Automatically replace AWS EC2 instances in AutoScaling groups with cheapest compatible spot instances https://autospotting.org
@/all Happy to announce that AutoSpotting now automatically handles the Spot Capacity Rebalancing events, to proactively drain workloads from terminating spot instances, without any required configuration changes.
Huge thanks to @gjmveloso from AWS for this contribution!
Huge thanks to @gjmveloso from AWS for this contribution!
As usual with the non-stable builds, this is experimental code, so please let us know if you notice any issues.
I'm actually working on CloudFormation simplifications as part of the StackSet change so I may take care of this myself. here are my plans for the next few days:
- extract the Manage ASG Python code into a separate file that could also be reused from Terraform
- look into using SAM for the CloudFormation stack, which should be able to also use this file
- update the Terraform code to match what we currently have in CloudFormation, leveraging recent TF features for creating the regional resources
@/all if anyone is bored over the weekend, I'd like to have a review of my recent CloudFormation stack code changes that greatly simplify the template code by removing most conditionals and make use of StackSets for deploying the regional resources instead of a custom resource backed by Lambda https://github.com/AutoSpotting/AutoSpotting/pull/449/files For even more review fun there are also a few code style changes on the Go code and a bunch of deletions of unused code :-)
next week I'll work to port all this stuff to Terraform, now that I understand again what's really going on there :-)
Hey !! Autospotting looks too awesome ! I was going through https://rancher.com/reducing-aws-spend and was wondering if Rancher still supports autospotting ?
but don't let that stop you from giving it a try, AutoSpotting improved a great deal since that blog post
Do you know if there's a tested use case of autospotting + k8s autoscaler ?
And I will test it on a project that is using it, I am sure that will help us bring down some $$s
so I'd recommend to use the CloudFormation until that's fixed, unless you want to figure it out and fix it yourself
Thanks for headsup, I just checked this with cloudformation and it looks there's something wrong the way it estimates the billing savings. I am seeing below log messages
SC:2020-11-26T05:33:00 2020/11/26 05:33:33 autoscaling.go:247: ap-south-1 eks-worker-asg-dev Skipping group, license limit reached: would reach estimated monthly savings of $2643.70 when processing this group, above the $1000 evaluation limit
I tested this on ASG which was costing me $100/mo running 2x
c5.large on-demand instances
Also, I kept the desired/min/max capacity to 2 instances on the ASG, however I am seeing 4 instances which are created out of which 2 of them are in ASG and 2 are not
those outside the ASG should be eventually swapped against those from the ASG, just give it some time. regarding the savings limit, I'm contemplating to remove it soon and bring back the no-limit but expiration within 30 or 60 days model that I had in the past, as it's too much hassle to maintain and update this savings computation logic to work with the new event-based execution mode
@/all reminder that the second Ask me Anything about AutoSpotting Zoom call is scheduled in about 45min, everyone is invited and can join at https://www.linkedin.com/events/askmeanythingzoomcallaboutautos6727895500017823744/
@/all I've just pushed a draft PR (AutoSpotting/terraform-aws-autospotting#37) that updates the Terraform code to match the latest CloudFormation template, please have a look and provide feedback. I successfully applied it but didn't yet test if AutoSpotting actually works with it. Testing it and any sort of feedback about it would be more than welcome
@aaratn try the latest build, I removed the savings limits logic and replaced with the expiration logic we had in the past
@aaratn in the latest version of the Terraform module I just released there's now also experimental Terraform native support for all the features available in CloudFormation, you may want to give that a try
But perhaps we could do the math and choose the GP3 automatically below a certain size threshold
yeah suppose so. looks like you have to have a GP2 volume at least 170 GiB before it can burst above 128 MiB/s, and larger than 334 GiB before it'll do 250 MiB/s all the time.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html
I think it makes sense to have the conversion automatically done based on a size threshold configurable by the user with a sane default: 170GB if you need to match the throughout and 1TB for IOPS
@/all experimental/WIP support for the automated storage upgrade from GP2->GP3 and IO1->IO2 EBS volume types is now available at AutoSpotting/AutoSpotting#436 Feel free to try it out and report issues