Instance Group auto-scaling based on Cloud Pub/Sub queue count
It would be great to have an auto-scaling metric for GCE Instance Groups (or k8s 'Jobs') based on a Cloud Pub/Sub's topic count. For asynchronous workloads, as tasks are written into the topic (queue), spin up/down workers from an instance group's template. Additionaly, as Cloud Functions become fully supported, more events will are hoping to move more workloads into a dynamicly scalling into Cloud Pub/Sub - Instance Group workflow.
zohaib butt commented
is there any solution around instances terminating while processing a message? we have long-running tasks in a MIG and want to find a way to prevent this. Was wondering if the stop/terminate request on an instance can be delayed until task is completed... any suggestions?
Zach Bjornson commented
This is sort of available now (https://cloudplatform.googleblog.com/2018/03/queue-based-scaling-made-easy-with-new-stackdriver-per-group-metrics.html), but seems to have the fatal flaw that instances can be terminated before they are done processing a message:
> Note that [num_undelivered_messages] exports the total number of messages in the queue, including messages that are currently being processed but that are not yet acknowledged. Using a metric that does not include the messages being processed is not recommended because such a metric can drop down to 0 when there is still work being done, which prompts autoscaling to scale down and possibly interrupt the actual work.
Kevin Moore commented
I believe this is what you want
Now in alpha