Specify properties that determine which instance to remove when autoscaling down group
I run an autoscale group with a minimum of 1 and a maximum of 6 instances. They have 6 cores and 16GB of memory. Each of them host their own Varnish cache in addition to running the main application code. We're not large enough to have a dedicated Varnish group. Ensuring that each machine is responsible for its own cache means there is no SPOF, and allows us to run A/B tests without corrupting a global cache.
Our services are very resource intensive, and a warmed cache significantly improves the user experience. Each instance is configured to peak around 86% memory usage and maintain that. When that usag level is reached, the given instance is very fast, because it is heavily cached.
I've noticed that when an autoscale group determines that an instance should be removed, it typically removes the ones that have the greatest memory usage. This could just be coincidental, but I've noticed the pattern. This is problematic. The instance that is targeted for removal, possibly because of its high memory usage, also happens to be the best performing instance, because it has cached most of its services. After a little while, a new instance is added to the group, but it has no cache, which means the overall experience for end users is actually degraded: the most-cached, best-performing instance was removed, and later, a cold-cache, least-performing instance was added.
Add the ability to specify certain properties that an instance must or must not have in order for it to be qualified for removal from a group. This would allow us, for example, to ensure that instances with the most cached content (highest memory usage) would not be targeted for removal; instead, the logic would look at the instances that have the least amount of cache data (memory usage), and even the date of creation (youngest instance in group), and then remove those instances instead. This would allow our services to remain heavily-cached, and ensure a more consistent experience for our end users.
I’ve confirmed with the team that a few options along these lines are under investigation.