Skip to content

Post 14.6 upgrade, sikekiq looping and fails to start and CICD tasks no longer run

Summary

Upgrading from 14.5.2 to 14.6.0 or to 14.6.1, it leads to sidekiq being unable to start as expected and looping permanently.

1 test was to upgrade from 14.5.2 to 14.6.0 during Christmas break => restore 14.5.2 as CI/CD tasks could no longer run

Just tested from 14.5.2 to 14.6.1 and no changes.

I use omnibus package on Debian 10.x server - all up to date.

Steps to reproduce

  • Having a 14.5.2 instance
  • Upgrade to 14.6.0/14.6.1
  • After upgrade step, siekiq looping indefinitely with logs below
  • CI/CD tasks no longer runs

What is the current bug behavior?

  • CI/CD tasks no longer starts.
  • Sidekiq components loops indefinitely

What is the expected correct behavior?

Sidekiq should run and CI/CD tasks to run accordingly

Relevant logs and/or screenshots

2022-01-05T08:48:01.504Z pid=7710 tid=d2m INFO: Listening on queues ["analytics_usage_trends_counter_job", "approve_blocked_pending_approval_users", "authorized_keys", "authorized_project_update:authorized_project_update_project_create", "authorized_project_update:authorized_project_update_project_group_link_create", "authorized_project_update:authorized_project_update_project_recalculate", "authorized_project_update:authorized_project_update_project_recalculate_per_user", "authorized_project_update:authorized_project_update_user_refresh_from_replica", "authorized_project_update:authorized_project_update_user_refresh_over_user_range", "authorized_project_update:authorized_project_update_user_refresh_with_low_urgency", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "background_migration", "bulk_import", "bulk_imports_entity", "bulk_imports_export_request", "bulk_imports_pipeline", "bulk_imports_relation_export", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "ci_delete_objects", "container_repository:cleanup_container_repository", "container_repository:container_expiration_policies_cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_note_diff_file", "cronjob:admin_email", "cronjob:analytics_usage_trends_count_job_trigger", "cronjob:authorized_project_update_periodic_recalculate", "cronjob:ci_archive_traces_cron", "cronjob:ci_delete_unit_tests", "cronjob:ci_pipeline_artifacts_expire_artifacts", "cronjob:ci_platform_metrics_update_cron", "cronjob:ci_schedule_delete_objects_cron", "cronjob:ci_stuck_builds_drop_running", "cronjob:ci_stuck_builds_drop_scheduled", "cronjob:clusters_integrations_check_prometheus_health", "cronjob:container_expiration_policy", "cronjob:database_batched_background_migration", "cronjob:database_drop_detached_partitions", "cronjob:database_partition_management", "cronjob:dependency_proxy_image_ttl_group_policy", "cronjob:environments_auto_delete_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:gitlab_service_ping", "cronjob:import_export_project_cleanup", "cronjob:import_stuck_project_import_jobs", "cronjob:issue_due_scheduler", "cronjob:issues_reschedule_stuck_issue_rebalances", "cronjob:jira_import_stuck_jira_import_jobs", "cronjob:loose_foreign_keys_cleanup", "cronjob:member_invitation_reminder_emails", "cronjob:metrics_dashboard_schedule_annotations_prune", "cronjob:namespaces_in_product_marketing_emails", "cronjob:namespaces_prune_aggregation_schedules", "cronjob:packages_composer_cache_cleanup", "cronjob:pages_domain_removal_cron", "cronjob:pages_domain_ssl_renewal_cron", "cronjob:pages_domain_verification_cron", "cronjob:partition_creation", "cronjob:personal_access_tokens_expired_notification", "cronjob:personal_access_tokens_expiring", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:releases_manage_evidence", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_unaccepted_member_invites", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_dispatch", "cronjob:requests_profiles", "cronjob:schedule_merge_request_cleanup_refs", "cronjob:schedule_migrate_external_diffs", "cronjob:ssh_keys_expired_notification", "cronjob:ssh_keys_expiring_soon_notification", "cronjob:stuck_ci_jobs", "cronjob:stuck_export_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "cronjob:update_container_registry_info", "cronjob:user_status_cleanup_batch", "cronjob:users_create_statistics", "cronjob:users_deactivate_dormant_users", "cronjob:x509_issuer_crl_check", "default", "delete_diff_files", "delete_merged_branches", "delete_stored_files", "delete_user", "dependency_proxy:purge_dependency_proxy_cache", "dependency_proxy_blob:dependency_proxy_cleanup_blob", "dependency_proxy_manifest:dependency_proxy_cleanup_manifest", "deployment:deployments_archive_in_project", "deployment:deployments_drop_older_deployments", "deployment:deployments_hooks", "deployment:deployments_link_merge_request", "deployment:deployments_update_environment", "design_management_copy_design_collection", "design_management_new_version", "destroy_pages_deployments", "detect_repository_languages", "disallow_two_factor_for_group", "disallow_two_factor_for_subgroups", "email_receiver", "emails_on_push", "environments_auto_stop", "environments_canary_ingress_update", "error_tracking_issue_link", "experiments_record_conversion_event", "expire_build_instance_artifacts", "export_csv", "external_service_reactive_caching", "file_hook", "flush_counter_increments", "gcp_cluster:cluster_configure_istio", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_patch_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_update_app", "gcp_cluster:cluster_upgrade_app", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_app_update", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:clusters_applications_activate_service", "gcp_cluster:clusters_applications_deactivate_service", "gcp_cluster:clusters_applications_uninstall", "gcp_cluster:clusters_applications_wait_for_uninstall_app", "gcp_cluster:clusters_cleanup_project_namespace", "gcp_cluster:clusters_cleanup_service_account", "gcp_cluster:wait_for_cluster_creation", "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_lfs_object", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_import_pull_request_merged_by", "github_importer:github_import_import_pull_request_review", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_lfs_objects", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_pull_requests_merged_by", "github_importer:github_import_stage_import_pull_requests_reviews", "github_importer:github_import_stage_import_repository", "gitlab_performance_bar_stats", "gitlab_shell", "group_destroy", "group_export", "group_import", "hashed_storage:hashed_storage_migrator", "hashed_storage:hashed_storage_project_migrate", "hashed_storage:hashed_storage_project_rollback", "hashed_storage:hashed_storage_rollbacker", "import_issues_csv", "incident_management:incident_management_add_severity_system_note", "incident_management:incident_management_pager_duty_process_incident", "incident_management:incident_management_process_alert_worker_v2", "integrations_create_external_cross_reference", "invalid_gpg_signature_update", "irker", "issuable_export_csv", "issuable_label_links_destroy", "issuables_clear_groups_issue_counter", "issue_placement", "issue_rebalancing", "issues_placement", "issues_rebalancing", "jira_connect:jira_connect_forward_event", "jira_connect:jira_connect_retry_request", "jira_connect:jira_connect_sync_branch", "jira_connect:jira_connect_sync_builds", "jira_connect:jira_connect_sync_deployments", "jira_connect:jira_connect_sync_feature_flags", "jira_connect:jira_connect_sync_merge_request", "jira_connect:jira_connect_sync_project", "jira_importer:jira_import_advance_stage", "jira_importer:jira_import_import_issue", "jira_importer:jira_import_stage_finish_import", "jira_importer:jira_import_stage_import_attachments", "jira_importer:jira_import_stage_import_issues", "jira_importer:jira_import_stage_import_labels", "jira_importer:jira_import_stage_import_notes", "jira_importer:jira_import_stage_start_import", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "merge_request_cleanup_refs", "merge_request_mergeability_check", "merge_requests_delete_source_branch", "merge_requests_handle_assignees_change", "merge_requests_resolve_todos", "metrics_dashboard_prune_old_annotations", "metrics_dashboard_sync_dashboards", "migrate_external_diffs", "namespaceless_project_destroy", "namespaces_invite_team_email", "namespaces_onboarding_issue_created", "namespaces_onboarding_pipeline_created", "namespaces_onboarding_progress", "namespaces_onboarding_user_added", "namespaces_process_sync_events", "new_issue", "new_merge_request", "new_note", "object_pool:object_pool_create", "object_pool:object_pool_destroy", "object_pool:object_pool_join", "object_pool:object_pool_schedule_join", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "package_repositories:packages_debian_generate_distribution", "package_repositories:packages_debian_process_changes", "package_repositories:packages_go_sync_packages", "package_repositories:packages_helm_extraction", "package_repositories:packages_maven_metadata_sync", "package_repositories:packages_nuget_extraction", "package_repositories:packages_rubygems_extraction", "packages_composer_cache_update", "pages", "pages_domain_ssl_renewal", "pages_domain_verification", "pages_transfer", "pages_update_configuration", "phabricator_import_import_tasks", "pipeline_background:archive_trace", "pipeline_background:ci_archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_background:ci_daily_build_group_report_results", "pipeline_background:ci_pending_builds_update_group", "pipeline_background:ci_pending_builds_update_project", "pipeline_background:ci_pipeline_artifacts_coverage_report", "pipeline_background:ci_pipeline_artifacts_create_quality_report", "pipeline_background:ci_pipeline_success_unlock_artifacts", "pipeline_background:ci_ref_delete_unlock_artifacts", "pipeline_background:ci_test_failure_history", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation:ci_external_pull_requests_create_pipeline", "pipeline_creation:create_pipeline", "pipeline_creation:merge_requests_create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default:ci_create_cross_project_pipeline", "pipeline_default:ci_create_downstream_pipeline", "pipeline_default:ci_drop_pipeline", "pipeline_default:ci_merge_requests_add_todo_when_build_fails", "pipeline_default:ci_pipeline_bridge_status", "pipeline_default:ci_retry_pipeline", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:ci_build_finished", "pipeline_processing:ci_build_prepare", "pipeline_processing:ci_build_schedule", "pipeline_processing:ci_initial_pipeline_process", "pipeline_processing:ci_resource_groups_assign_resource_from_resource_group", "pipeline_processing:pipeline_process", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "post_receive", "process_commit", "project_cache", "project_daily_statistics", "project_destroy", "project_export", "project_service", "projects_git_garbage_collect", "projects_post_creation", "projects_process_sync_events", "projects_schedule_bulk_repository_shard_moves", "projects_update_repository_storage", "prometheus_create_default_alerts", "propagate_integration", "propagate_integration_group", "propagate_integration_inherit", "propagate_integration_inherit_descendant", "propagate_integration_project", "reactive_caching", "rebase", "releases_create_evidence", "remote_mirror_notification", "repository_check:repository_check_batch", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_cleanup", "repository_fork", "repository_import", "repository_remove_remote", "repository_update_remote_mirror", "self_monitoring_project_create", "self_monitoring_project_delete", "service_desk_email_receiver", "snippets_schedule_bulk_repository_shard_moves", "snippets_update_repository_storage", "system_hook_push", "tasks_to_be_done_create", "todos_destroyer:todos_destroyer_confidential_issue", "todos_destroyer:todos_destroyer_destroyed_designs", "todos_destroyer:todos_destroyer_destroyed_issuable", "todos_destroyer:todos_destroyer_entity_leave", "todos_destroyer:todos_destroyer_group_private", "todos_destroyer:todos_destroyer_private_features", "todos_destroyer:todos_destroyer_project_private", "unassign_issuables:members_destroyer_unassign_issuables", "update_external_pull_requests", "update_highest_role", "update_merge_requests", "update_namespace_statistics:namespaces_root_statistics", "update_namespace_statistics:namespaces_schedule_aggregation", "update_project_statistics", "upload_checksum", "web_hook", "web_hooks_destroy", "web_hooks_log_execution", "wikis_git_garbage_collect", "x509_certificate_revoke"]
{"severity":"INFO","time":"2022-01-05T08:48:01.506Z","message":"Booting Sidekiq 6.3.1 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/opt/gitlab/redis/redis.socket\", :namespace=\u003e\"resque:gitlab\"}"}
{"severity":"INFO","time":"2022-01-05T08:48:02.080Z","message":"GitLab reliable fetch activated!"}
{"severity":"INFO","time":"2022-01-05T08:48:02.109Z","message":"Cron Jobs - add job with name: stuck_ci_jobs_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.110Z","message":"Cron Jobs - add job with name: pipeline_schedule_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.112Z","message":"Cron Jobs - add job with name: expire_build_artifacts_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.113Z","message":"Cron Jobs - add job with name: ci_pipelines_expire_artifacts_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.115Z","message":"Cron Jobs - add job with name: ci_schedule_delete_objects_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.116Z","message":"Cron Jobs - add job with name: environments_auto_stop_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.118Z","message":"Cron Jobs - add job with name: environments_auto_delete_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.119Z","message":"Cron Jobs - add job with name: repository_check_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.121Z","message":"Cron Jobs - add job with name: admin_email_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.122Z","message":"Cron Jobs - add job with name: personal_access_tokens_expiring_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.124Z","message":"Cron Jobs - add job with name: personal_access_tokens_expired_notification_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.126Z","message":"Cron Jobs - add job with name: repository_archive_cache_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.127Z","message":"Cron Jobs - add job with name: import_export_project_cleanup_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.128Z","message":"Cron Jobs - add job with name: ci_archive_traces_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.130Z","message":"Cron Jobs - add job with name: requests_profiles_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.131Z","message":"Cron Jobs - add job with name: remove_expired_members_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.133Z","message":"Cron Jobs - add job with name: remove_expired_group_links_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.134Z","message":"Cron Jobs - add job with name: remove_unaccepted_member_invites_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.136Z","message":"Cron Jobs - add job with name: prune_old_events_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.137Z","message":"Cron Jobs - add job with name: trending_projects_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.139Z","message":"Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.140Z","message":"Cron Jobs - add job with name: import_stuck_project_import_jobs"}
{"severity":"INFO","time":"2022-01-05T08:48:02.142Z","message":"Cron Jobs - add job with name: jira_import_stuck_jira_import_jobs"}
{"severity":"INFO","time":"2022-01-05T08:48:02.143Z","message":"Cron Jobs - add job with name: stuck_export_jobs_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.145Z","message":"Cron Jobs - add job with name: gitlab_service_ping_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.146Z","message":"Cron Jobs - add job with name: stuck_merge_jobs_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.148Z","message":"Cron Jobs - add job with name: pages_domain_verification_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.149Z","message":"Cron Jobs - add job with name: pages_domain_removal_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.151Z","message":"Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.152Z","message":"Cron Jobs - add job with name: issue_due_scheduler_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.154Z","message":"Cron Jobs - add job with name: metrics_dashboard_schedule_annotations_prune_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.155Z","message":"Cron Jobs - add job with name: schedule_migrate_external_diffs_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.157Z","message":"Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.158Z","message":"Cron Jobs - add job with name: container_expiration_policy_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.160Z","message":"Cron Jobs - add job with name: image_ttl_group_policy_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.161Z","message":"Cron Jobs - add job with name: x509_issuer_crl_check_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.163Z","message":"Cron Jobs - add job with name: users_create_statistics_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.164Z","message":"Cron Jobs - add job with name: authorized_project_update_periodic_recalculate_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.166Z","message":"Cron Jobs - add job with name: update_container_registry_info_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.167Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_manager"}
{"severity":"INFO","time":"2022-01-05T08:48:02.169Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_dropper"}
{"severity":"INFO","time":"2022-01-05T08:48:02.179Z","message":"Cron Jobs - add job with name: ci_platform_metrics_update_cron_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.180Z","message":"Cron Jobs - add job with name: analytics_usage_trends_count_job_trigger_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.182Z","message":"Cron Jobs - add job with name: member_invitation_reminder_emails_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.183Z","message":"Cron Jobs - add job with name: schedule_merge_request_cleanup_refs_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.185Z","message":"Cron Jobs - add job with name: manage_evidence_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.186Z","message":"Cron Jobs - add job with name: user_status_cleanup_batch_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.188Z","message":"Cron Jobs - add job with name: ssh_keys_expired_notification_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.189Z","message":"Cron Jobs - add job with name: namespaces_in_product_marketing_emails_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.191Z","message":"Cron Jobs - add job with name: ssh_keys_expiring_soon_notification_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.192Z","message":"Cron Jobs - add job with name: users_deactivate_dormant_users_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.194Z","message":"Cron Jobs - add job with name: ci_delete_unit_tests_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.195Z","message":"Cron Jobs - add job with name: batched_background_migrations_worker"}
{"severity":"INFO","time":"2022-01-05T08:48:02.197Z","message":"Cron Jobs - add job with name: issues_reschedule_stuck_issue_rebalances"}
{"severity":"INFO","time":"2022-01-05T08:48:02.198Z","message":"Cron Jobs - add job with name: clusters_integrations_check_prometheus_health_worker"}
{"severity":"WARN","time":"2022-01-05T08:48:03.327Z","message":"Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses."}
{"severity":"WARN","time":"2022-01-05T08:48:03.358Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2022-01-05T08:48:03.473Z","message":"Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines."}
{"severity":"WARN","time":"2022-01-05T08:48:03.715Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2022-01-05T08:48:03.716Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2022-01-05T08:48:03.716Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2022-01-05T08:48:03.734Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2022-01-05T08:48:04.133Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2022-01-05T08:48:04.133Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2022-01-05T08:48:05.265Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2022-01-05T08:48:05.572Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2022-01-05T08:48:05.572Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2022-01-05T08:48:07.096Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2022-01-05T08:48:07.096Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2022-01-05T08:48:07.100Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2022-01-05T08:48:07.213Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2022-01-05T08:48:07.336Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2022-01-05T08:48:07.743Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2022-01-05T08:48:07.743Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2022-01-05T08:48:07.743Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2022-01-05T08:48:07.744Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2022-01-05T08:48:07.744Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2022-01-05T08:48:07.744Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2022-01-05T08:48:07.744Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2022-01-05T08:48:07.744Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2022-01-05T08:48:07.745Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2022-01-05T08:48:07.745Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2022-01-05T08:48:07.745Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2022-01-05T08:48:07.745Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"WARN","time":"2022-01-05T08:48:07.795Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2022-01-05T08:48:11.019Z","message":"Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system."}
{"severity":"INFO","time":"2022-01-05T08:48:17.313Z","message":"Booted Rails 6.1.4.1 application in production environment"}
{"severity":"INFO","time":"2022-01-05T08:48:17.313Z","message":"Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]"}
{"severity":"INFO","time":"2022-01-05T08:48:17.313Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2022-01-05T08:48:17.313Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
{"severity":"INFO","time":"2022-01-05T08:48:18.005Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2022-01-05T08:48:18.986Z","message":"Starting cluster with 1 processes"}

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Debian 10
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.6
Redis Version:	6.0.16
Git Version:	2.33.1.
Sidekiq Version:6.3.1
Go Version:	unknown

GitLab information
Version:	14.6.1
Revision:	661d663ab2b
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://code.corp.com
HTTP Clone URL:	https://code.corp.com/some-group/some-project.git
SSH Clone URL:	git@code.corp.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.22.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... Checking code@corp.com
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/1 ... yes
2/6 ... yes
12/9 ... yes
13/13 ... yes
12/18 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
12/30 ... yes
12/33 ... yes
12/35 ... yes
12/36 ... yes
2/38 ... yes
12/39 ... yes
2/40 ... yes
12/41 ... yes
12/42 ... yes
13/43 ... yes
13/44 ... yes
2/45 ... yes
2/46 ... yes
12/48 ... yes
2/50 ... yes
12/51 ... yes
12/52 ... yes
12/53 ... yes
12/54 ... yes
2/55 ... yes
51/57 ... yes
56/58 ... yes
12/59 ... yes
12/61 ... yes
12/63 ... yes
59/64 ... yes
12/66 ... yes
12/69 ... yes
12/70 ... yes
12/71 ... yes
12/72 ... yes
12/73 ... yes
60/74 ... yes
12/75 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 4
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Edited by Nicolas Steinmetz