Skip to content

Include logs when a failed review-app occurred

David Dieulivol requested to merge ddieulivol-show_logs_of_failed_review_apps into master

What does this MR do and why?

Add logs for the webservice and database pods when a review-deploy failed.

This is so that anybody wanting to debug a broken review-app doesn't have to leave the failed job page (currently, you'd have to click on many places for it to happen).

How to set up and validate locally

Here's what can be seen if we are working from the review-smriti-378-7etuge namespace, which as a few broken pods:

$ kubectl logs -l app=webservice --all-containers --namespace "${namespace}"
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:43:47Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:43:57Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:07Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:17Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:27Z","uri":""}
{"correlation_id":"","duration_ms":2,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:37Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:47Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:44:57Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:45:07Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-11-24T07:45:17Z","uri":""}
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
+ cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret
'/init-config/gitlab-workhorse/secret' -> '/init-secrets-workhorse/gitlab-workhorse/secret'
+ mkdir -p /init-secrets-workhorse/redis
+ cp -v -r -L /init-config/redis/redis-password /init-secrets-workhorse/redis/
'/init-config/redis/redis-password' -> '/init-secrets-workhorse/redis/redis-password'
+ '[' -d /init-config/minio ]
+ mkdir -p /init-secrets-workhorse/minio
+ cp -v -r -L /init-config/minio/accesskey /init-config/minio/secretkey /init-secrets-workhorse/minio/
'/init-config/minio/accesskey' -> '/init-secrets-workhorse/minio/accesskey'
'/init-config/minio/secretkey' -> '/init-secrets-workhorse/minio/secretkey'
Database Schema - main (gitlabhq_production) - current: 0, codebase: 20221121180138
NOTICE: Database has not been initialized yet.
Checking: main
Database Schema - main (gitlabhq_production) - current: 0, codebase: 20221121180138
NOTICE: Database has not been initialized yet.
Checking: main
Database Schema - main (gitlabhq_production) - current: 0, codebase: 20221121180138
NOTICE: Database has not been initialized yet.
Checking: main
Database Schema - main (gitlabhq_production) - current: 20221121180138, codebase: 20221121180138
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:31:in `dispatch'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:25:in `start'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:48:in `block in <top (required)>'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /usr/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:36:in `<top (required)>'
	from /srv/gitlab/bin/bundle:5:in `load'
	from /srv/gitlab/bin/bundle:5:in `<main>'

and

 $ kubectl logs -l app=postgresql --all-containers                           
2022-11-23 19:35:41.277 GMT [289] ERROR:  relation "postgres_partitioned_tables" does not exist at character 85
2022-11-23 19:35:41.277 GMT [289] STATEMENT:  /*application:web,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'batched_background_migration_job_transition_logs')) LIMIT 1
2022-11-23 19:35:41.278 GMT [289] ERROR:  relation "postgres_partitioned_tables" does not exist at character 85
2022-11-23 19:35:41.278 GMT [289] STATEMENT:  /*application:web,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'incident_management_pending_alert_escalations')) LIMIT 1
2022-11-23 19:35:41.280 GMT [289] ERROR:  relation "postgres_partitioned_tables" does not exist at character 85
2022-11-23 19:35:41.280 GMT [289] STATEMENT:  /*application:web,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'incident_management_pending_issue_escalations')) LIMIT 1
2022-11-23 19:35:41.281 GMT [289] ERROR:  relation "postgres_partitioned_tables" does not exist at character 85
2022-11-23 19:35:41.281 GMT [289] STATEMENT:  /*application:web,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'security_findings')) LIMIT 1
2022-11-23 19:35:41.283 GMT [289] ERROR:  relation "postgres_partitioned_tables" does not exist at character 85
2022-11-23 19:35:41.283 GMT [289] STATEMENT:  /*application:web,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'verification_codes')) LIMIT 1

Which already tells us that the database has some issues 🎉

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports