Skip to content

[Docker Machine Executor] Provide controls for bursty, unpredictable load scenarios

Description

Context: Docker+Machine Executor

The current autoscale algorithm relies purely on meeting the IdleCount, and adds machines at a rate set by the minimum of MaxGrowthRate or IdleCount. Dynamic adjustment of these values can only be done over pre-defined time period sub-configurations, but not based on the incoming, pending job load.

In the case of a workload where the periods of busy workloads cannot be defined, and a lot of jobs get generated for execution, there are a few problems that crop up:

  • New host provision rate is controlled by IdleCount as defined above, and does not try to provision a larger amount of machines quicker based on the accumulated load.
  • The use of IdleCount to define the provisioning rate causes an increase in costs, as after the load period passes a raised IdleCount keeping as many hosts running forever.

Proposal

Provide an alternative configuration to IdleCount to express a value that controls rate of new host provisions during an unpredicted high jobs load event.

The provision rate control provided should not result in a number of idle hosts left running after the job load event has passed.

Links to related issues and merge requests / references

Edited by Harsh Chouraria