Some migrations create errors on decomposed databases

See discussion below.

To replicate:

1. Migrate from single database to two databases
2. Do not truncate
3. Run the migrations below

The issue stems from us enforcing that

1. Data migrations must be run on one database only - main in this case
2. So data on ci for merge_request_user_mentions is not fixed
3. When unique index is created for merge_request_user_mentions on main, it creates successfully
4. When unique index is created for merge_request_user_mentions on ci, it fails

/cc @Kras @stomlinson @ayufan @DylanGriffith @OmarQunsulGitlab @rutgerwessels

Workaround

After migrating to decomposed, truncate the legacy data

Proposal

1. Update docs. Advise decomposed instances to truncate before next upgrade.
2. Add a upgrade check to warn if data is not truncated.

Original report

The following discussion from !113928 (merged) should be addressed:

Click to expand

• @smcgivern started a discussion: (+7 comments)

  @dfrazao-gitlab @krasio I see that this is present on this migration but not the following one. I then get an error when running against my CI DB:

  ci: -- add_index("merge_request_user_mentions", :merge_request_id, {:unique=>true, :name=>"merge_request_user_mentions_on_mr_id_index_convert_to_bigint", :where=>"note_id_convert_to_bigint IS NULL", :algorithm=>:concurrently})
  rake aborted!
  StandardError: An error has occurred, all later migrations canceled:
  
  PG::UniqueViolation: ERROR:  could not create unique index "merge_request_user_mentions_on_mr_id_index_convert_to_bigint"
  DETAIL:  Key (merge_request_id)=(26) is duplicated.

  And this is because note_id_convert_to_bigint is full of NULLs:

  gitlabhq_development_ci=# SELECT merge_request_id, note_id, note_id_convert_to_bigint FROM merge_request_user_mentions;
   merge_request_id | note_id | note_id_convert_to_bigint 
  ------------------+---------+---------------------------
                 26 |     825 |                          
                 26 |     827 |                          
                 71 |         |                          
                 72 |         |                          
                 73 |         |                          
                 74 |         |                          
                 75 |         |                          
                 76 |         |                          
                 77 |         |                          
                 78 |         |                          
  (10 rows)

  On my main DB, this is all fine:

  gitlabhq_development=# SELECT merge_request_id, note_id, note_id_convert_to_bigint FROM merge_request_user_mentions;
   merge_request_id | note_id | note_id_convert_to_bigint 
  ------------------+---------+---------------------------
                 26 |     825 |                       825
                 26 |     827 |                       827
                 71 |         |                          
                 72 |         |                          
                 73 |         |                          
                 74 |         |                          
                 75 |         |                          
                 76 |         |                          
                 77 |         |                          
                 78 |         |                          
  (10 rows)

  And the index gets created successfully.