Git sometimes reports branch does not exist for target branch push option - possible race condition

Summary

Sometimes, when using push options to create a merge request with a specified target branch, Git reports that the branch does not exist even though it definitely does.

The problem was discovered by an automated E2E QA test. See #254951 (closed)

Steps to reproduce

1. Use a script or run the automated test to quickly execute the commands.
2. Push a new branch.
3. Use push options to create a merge request with a specific target branch.
4. Repeat until it fails

For example, the automated test uses the Git CLI to create a branch and push it

git checkout -b "push-options-test-target-b5dc59c51402f7b7"
# add and commit files
git push

And then confirms that the branch exists by cloning the project in a separate directory and checking out the branch.

$ cd /new/tmp/dir
$ git clone ...
Cloning into '.'...

$ git checkout  "push-options-test-target-b5dc59c51402f7b7"
Switched to a new branch 'push-options-test-target-b5dc59c51402f7b7'

It then tries to use push options to create a merge request against a specific target branch:

git push -o merge_request.create \
         -o merge_request.title="MR push options test 88af550668bb1657" \
         -o merge_request.target="push-options-test-target-b5dc59c51402f7b7" \
         http://root@gitlab-ee-1f76daa1.test/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-07-08-cf699eb93bee1e59/merge-request-push-options-c7769e368b0742ae.git push-options-test-6a34683e20c3c53b:push-options-test-6a34683e20c3c53b

remote: ========================================================================        
remote: 
remote:              WARNINGS: Error encountered with push options        
remote:     'merge_request.create' 'merge_request.title=MR push options test        
remote:                            88af550668bb1657'        
remote:    'merge_request.target=push-options-test-target-b5dc59c51402f7b7':        
remote:     Branch push-options-test-target-b5dc59c51402f7b7 does not exist
remote: 
remote: ========================================================================        
remote: 
remote: 
remote: To create a merge request for push-options-test-6a34683e20c3c53b, visit:        
remote:   http://gitlab-ee-1f76daa1.test/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-07-08-cf699eb93bee1e59/merge-request-push-options-c7769e368b0742ae/-/merge_requests/new?merge_request%5Bsource_branch%5D=push-options-test-6a34683e20c3c53b
remote: 

The automated QA test can be run in a loop with the following command:

GITLAB_QA_LOOP_RUNNER_MINUTES=60 gitlab-qa Test::Instance::Image EE --loop -- qa/specs/features/api/3_create/merge_request/push_options_target_branch_spec.rb

What is the current bug behavior?

The merge request is sometimes not created with the correct target branch.

What is the expected correct behavior?

The merge request is always created with the correct target branch.