Support for Custom Scheduler
Description
What are we doing
We are using GitLab Runner in an EKS build cluster on AWS - working fine
- Compile, build and test artifacts in a continuous way (CI/CD)
- Test continuously the complete landscape using automated integration test and end-2-end tests. As we are working mainly in daylight hours, we are using the cluster-autoscaler to expand and shrink the cluster as needed.
Issue we are facing
The default EKS kube-scheduler uses a equal distribution mechanism to schedule pods on least utilized instances. That is fine as long as the cluster is small (few worker nodes). At the moment, the kubernetes cluster scales up, but it does not shrink down again. That is because of two things:
- A build job is per definition not able to be scheduled to another node at runtime (it has to start again in most cases)
- The jobs are evenly distributed across the cluster nodes and the autoscaler decides against scaling down, because the jobs can not be scheduled elsewhere.
Proposal
Kubernetes has the ability to deploy custom kube-schedulers, see https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ . Here it is possible to give the scheduler the hint to use a "new" priority, e.g. "MostRequestedPriority". That would in consequence mean that the scheduler tries to utilize the nodes to a maximum and not evenly distribute pods across the cluster. And no longer used nodes would be scaled down - far faster than in the default configuration that kubernetes uses and the gitlab runner utilizes.
Is it possible to extend the kubernetes executor with a configuration item, to use a custom scheduler? Result should be, that the kubernetes configuration flag ist given to all created pods (coordinator and job):
spec:
schedulerName: <CONFIGURE-ME>
e.g.
spec:
schedulerName: my-scheduler
I am far from being a Go Expert, but it seems that the place to do that is https://gitlab.com/gitlab-org/gitlab-runner/blob/master/executors/kubernetes/executor_kubernetes.go#L522
Thanks for reading so far - if I have overlooked something or that is already working, please feel free to advise me!
If you have questions, feel free to contact me.