Add build trace chunks migration duration histogram
Description
Current architecture of cloud native build logs assumes that build logs have to be migrated to a persistent store before we remove them from Redis. This is important, because otherwise we might loose data.
Data migration to object storage works in a way that we tell a runner to wait until we migrate a build log to object storage. This operation might take a while, but our method for measuring this today is not very accurate. It is more/less visible on the graph below - "accepted" operation is when we tell a runner to wait, and "finalized" operation happens when everything has been migrated to object storage.
The problem with this is that this is merely a counter, and we can only see approximation of delay knowing the runner polling / retry interval.
Proposal
We should add a build trace chunks migration histogram to collect much more relevant and accurate data.
Credits for this idea go to @ayufan.