Avoid unique constrain violation when loading existing build pending state
Description
The increased error rate has been reported regarding the unique constraint violation on ci_build_pending_states
table.
We identified that this happens by design, because the create_or_find_by
method depends on always inserting a record and catching ActiveRecord::RecordNotUnique
to load a resource.
def create_or_find_by(attributes, &block)
transaction(requires_new: true) { create(attributes, &block) }
rescue ActiveRecord::RecordNotUnique
find_by!(attributes)
end
See discussion in gitlab-com/gl-infra/production#2937 (comment 438995893)
Proposal
In order to reduce the noise generated by the database errors we can use find_or_create_by
method instead, followed by catching unique constraint violation. This should work better because we know that in at least 50% the pending state is going to exist, so there is no point in inserting it.
The race condition is also not very likely because runners update a build state sequentially using an exponential backoff mechanism.