PoC: Project-level Pipeline job sempahore
What does this MR do?
PoC for #15536 (closed), therefore it's not for merge
Technical proposal
- Introduce new tables
ci_project_semaphores
andci_job_locks
to track CI semaphores status. -
Ci::ProjectSemaphore
model is for managing all semaphores under the project. -
Ci::JobLock
model is for tracking the lock status of the job under a semaphore. -
job_lock
is created when a pipeline is created. Lockable jobs are executed by the order ofjob_lock.id
. (nearly equal to pipeline.id order) - If a job is lockable, the system delegates the operation to
job_lock
for controlling its status with taking account of semaphore. - When a job finished, it released a obtained lock. The system finds a next blocked job, updates the status to
locking
. - The system supports predefined variables for lock key. For example,
lock: $CI_COMMIT_REF_NAME
is translated tolock: master
. - We preserve the history of locked jobs. This is useful for investigating bottle neck of CI.
Concerns
- We should use compound status -
blocked
. Currently,Ci::Status
is super inefficient. - Any race conditions?
- Any edge cases?
- Potential deadlock?
- Performance concern?
Sample gitlab-ci.yml
job:
script: echo 'a'
lock: test
TODO
-
YAML parse && persist -
Present the extended status -
Update description -
Use compound status
Edited by Shinya Maeda