Autoscaling GitLab CI jobs on AWS Fargate (MVC)
Problem to Solve
Per the latest Datadog market analysis, “among companies running containers in AWS, 19 percent now use AWS Fargate, up from 5 percent a year earlier. Fargate's rapid growth has helped ECS keep pace with the continuing adoption of Kubernetes in AWS environments.”
Also, as illustrated in the comments on this issue over the past two years, the need to use AWS Fargate for running GitLab CI jobs is of high interest to the GitLab user community.
Proposal
Implement a solution that enables users to use AWS Fargate and specifically the AWS ECS Fargate launch type to automatically launch and scale containers that will execute the jobs in the user’s GitLab CI/CD pipeline.
MVC Scope
-
This feature will enable a GitLab Runner configured as a Runner Manager that is installed on an AWS EC2 instance to automatically create containers on AWS Fargate for executing the GitLab CI jobs.
-
A GitLab runner, configured as a Runner Manager, is able to spawn a Fargate task when a job arrives.
GitLab Runner Manager on EC2 instance
Documentation Draft (Autoscaling GitLab Runners on AWS Fargate)
With Autoscaling of GitLab Runners on AWS fargate, new Docker containers are automatically launched on your AWS Fargate cluster when needed to execute a CI/CD job in your GitLab pipeline.
Features
- Automatically responds to CI/CD job load by increasing and decreasing containers on AWS fargate.
- Containers on AWS Fargate are spawned by the GitRunner Manager and are deleted when no jobs are running.
- Each job in a pipeline will trigger the creation of new container on AWS Fargate.
Getting Started
- Create a new Amazon Web Services EC2 instance that will host the GitLab Runner Manager.
- Install GitLab Runner on the EC2 instance. Reference
- Configure the GitLab Runner to use the custom executor and the driver for AWS Fargate.
- Create an AWS Fargate cluster. This cluster will need network connectivity to the EC2 instance that is hosting the Runner Manager. A basic option is to create the Fargate cluster on the same Amazon Virtual Private Cloud (VPC) as the EC2 instance.
- Create a task definition that will be used to run the build job. For the MVC the goal is that GitLab will provide getting started task definition templates.
Items to Note (revised April 20 2020):
- Services won't be ready for the MVC.
- AWS Fargate does not support the creation of Docker images using docker build. It is possible to use other tooling for creation of Docker images. https://github.com/aws/containers-roadmap/issues/95
- Friday April 17: MVC binaries are now published and ready for testing by engineering. Plan for week of April 20 is to continue to test and iterate on the current code base.
User Requirements
| Requirement | in scope for MVC? | NOTES |
|---|---|---|
| Dynamically bring up containers on AWS Fargate when the pipeline needs to run a job. | Yes | |
| Pipeline automatically “takes the GitLab Runner down if there are no pending jobs” | NO |
Implementation Tasks for MVC
-
Create GitLab Runner AWS Fargate driver. The driver is responsible for calling the AWS Fargate RunTask action with LaunchType "FARGATE" to create and run a Docker container on ECS. The driver will be based on the work done for the custom executor. -
Create template for AWS Fargate task definition. -
Test performance of Runner autoscaling on Fargate. -
Create user documentation.