Log extra metadata sidekiq done for ElasticIndexBulkCronWorker
What does this MR do?
This introduces a generic worker capability
ApplicationWorker#log_extra_metadata_on_done(key, value)
to add
metadata that will later be logged with the done job payload. When
called this method will add the key/value to a Hash
which is then
pushed all the way up the middleware to the StructuredLogger
and
amended to the done log statement.
This MR also uses this functionality in ElasticIndexBulkCronWorker.
This is an improved way of doing the same thing as was implemented in !30265 (merged) which allows easily correlating the duration of a job with the size of the job being processed. This is better because sidekiq done logs are already broken out in detail for DB time, CPU time, Redis time and so we'd prefer to build on that existing behaviour when doing our analysis.
Screenshots
{
"severity": "INFO",
"time": "2020-05-07T05:20:09.978Z",
"queue": "cronjob:elastic_index_bulk_cron",
"args": [],
"class": "ElasticIndexBulkCronWorker",
"retry": false,
"backtrace": true,
"queue_namespace": "cronjob",
"jid": "214c52bf30bcc32e503e2a16",
"created_at": "2020-05-07T05:20:09.200Z",
"meta.caller_id": "Cronjob",
"correlation_id": "2395e4e2ff4d23078d43f76e32011e94",
"uber-trace-id": "51ce0edd1f327bd4:51ce0edd1f327bd4:0:1",
"enqueued_at": "2020-05-07T05:20:09.202Z",
"pid": 55633,
"message": "ElasticIndexBulkCronWorker JID-214c52bf30bcc32e503e2a16: done: 0.774345 sec",
"job_status": "done",
"scheduling_latency_s": 0.001691,
"redis_calls": 5,
"redis_duration_s": 0.00218,
"duration_s": 0.774345,
"cpu_s": 0.266514,
"completed_at": "2020-05-07T05:20:09.978Z",
- "db_duration_s": 0
-}
+ "db_duration_s": 0,
+ "extra.elastic_index_bulk_cron_worker.records_count": 15
+}
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team