Skip to content

Maintenance mode status can not be changed

Summary

The "There was an error updating the Maintenance Mode Settings" error happens when trying to change maintenance mode settings in the Admin area.

image.png

Steps to reproduce

The following steps reproduce the problem in the customer's instance that reported this issue.

  • Enable maintenance mode with the banner message
  • Click Tanuki icon
  • Visit the Admin area again
  • Try to disable the maintenance mode

I was unable to reproduce the issue using this exact same steps, but was able to reproduce the problem using the following steps.

(Based on how the below logs and screenshots show, I believe that the root cause of these issues are the same, bugs when the frontend makes API calls.)

  • Enable maintenance mode with the banner message
  • Refresh the page
  • Try to change banner message

What is the current bug behavior?

Can not change the maintenance mode settings.

What is the expected correct behavior?

Can change the maintenance mode settings.

Relevant logs and/or screenshots

Video recording from my sandbox

screenshot 2023-11-20 11.03.46.mov

image.png

image.png

ActiveRecord::NotNullViolation (PG::NotNullViolation: ERROR:  null value in column "maintenance_mode" of relation "application_settings" violates not-null constraint
{"time":"2023-11-20T02:59:37.953Z","severity":"INFO","duration_s":0.08914,"db_duration_s":0.00385,"view_duration_s":0.08529,"status":500,"method":"PUT","path":"/api/v4/application/settings","params":[{"key":"maintenance_mode","value":""},{"key":"maintenance_mode_message","value":"vvvvaaaa"}],"host":"<masked>","remote_ip":"113.32.144.145, 172.31.58.51, 113.32.144.145","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36","route":"/api/:version/application/settings","user_id":1,"username":"root","exception.class":"ActiveRecord::NotNullViolation","exception.message":"PG::NotNullViolation: ERROR:  null value in column \"maintenance_mode\" of relation \"application_settings\" violates not-null constraint\nDETAIL:  Failing row contains (1, 100000, f, t, null, 2023-10-02 11:58:56.274566, 2023-11-20 02:59:37.933505, null, 2, null, --- []\n, t, 100, 0, 0, t, null, 10080, ---\n- github\n- bitbucket\n- bitbucket_server\n- fogbugz\n- git\n- gi..., null, t, 100, null, 100, f, 48, f, localhost, 16, 10, 10, f, 8089, f, 15, f, 0, t, null, 1, null, NfthCxuJUS4hjEspDxPv, 5, , f, f, f, ---\n- default\n, null, t, , , , , 0, 0, 0, 0, t, t, 10, 50, 200, t, null, f, 0, 0, 0, 10, 3600, f, 30 days, http://localhost:9200, f, us-east-1, null, 10, fd376e59-637c-4e55-ac64-0aadcf88e917, 1.0, 2097152, f, 300, 30, 15, t, t, f, null, f, null, null, t, t, t, t, f, 3600, 3600, f, 7200, 3600, f, 7200, 3600, 55, 30, 10, t, t, t, null, f, null, null, t, null, 0.5, null, null, null, null, null, null, f, null, f, f, f, null, null, f, t, null, null, null, 204800, null, users.noreply.ec2-34-224-1-192.compute-1.amazonaws.com, t, |jhgJUh63pVzgs2zvDSkQZy94+ukeZ0sP48OEwSN8g1pq6RYC19678bbbf76f, 0, 0, f, 2, null, f, 0.0.0.0/0, ::/0, 5, 1, null, null, null, t, f, f, f, f, /-/grafana, f, {}, 300, f, t, null, f, null, null, null, null, null, null, f, 10, 60, {/users/password,/users/sign_in,/api/v3/session.json,/api/v3/ses..., f, 3600, 3600, 3, 3, null, 7, null, f, null, null, null, null, null, 2023-10-02 11:58:55.827214+00, null, f, null, t, 52428800, 8, null, null, null, null, null, null, null, null, null, null, null, null, f, f, null, 0, 10, 10, f, f, f, f, , t, f, 300, null, t, , , {}, null, f, f, {\"default\": 100}, 0, {}, t, null, 6, 6, 1, 6, 6, 1, null, vvvvaaaa, 52428800, 1024, f, 250, null, f, 0, f, https://gitpod.io/, null, t, https://docs.gitlab.com, f, oEgcmqA64dQUjf7RJIUEdKtceGhVKLxMjlGjsPXjsojRVLMPvkLE9lWQ5WKj\nTAP..., sZ8BUY8PbTruKWK9\n, 4, f, f, f, f, f, null, null, null, f, null, null, null, null, null, null, f, glpat-, null, f, 200, 15, t, 300, {}, {\"bpmn\": false, \"blockdiag\": false, \"excalidraw\": false}, t, null, f, f, f, null, null, null, null, 800, 15, 1000, 15, f, f, f, 0, null, null, f, null, null, null, 50000, 1000, {project,group}, null, null, f, f, null, null, t, 125, 15, 500, 15, f, f, 1048576, 100, 1000, 60, f, t, f, 3600, 3600, 200, 1, 100000, 0, t, 1800, 3600, f, 3600, 3600, f, 2, null, null, null, f, f, null, null, null, null, null, [], 2, 100, 3, 50, 300, free, 2022-01-23 00:00:00+00, null, null, null, 0, 0, 300, {}, t, 30, 10, null, null, null, null, https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-runner/re..., f, f, f, f, null, null, null, null, null, f, 2, 0, 1, t, t, client, 0, null, null, 1800, 600, 0, f, null, null, null, null, null, null, null, , 0.50, f, f, 0, 0, f, null, null, null, null, f, null, {}, |msLN/DAe+5YhdhXBQBG+IxHiXMRkK45kk2jQyaP2kQL8IeyCec3e744e0e27, f, 2, 90, f, 0, null, null, null, f, 0, t, f, f, f, null, f, 90, 7, f, 0, f, en, f, null, null, null, null, f, 0, t, t, f, f, 1, 50, 5000000, 100000, null, f, {}, f, t, 400, f, null, f, f, t, null, null, 2, null, null, f, {1,2,3,4,5,6,7,8,9,10,11,12}, 150, t, null, null, f, https://embed.diagrams.net, t, f, 300, 2, null, null, null, null, null, null, t, f, 7, {}, 600, f, 2, 1, t, 5120, 10240, {}, 25600, 0, null, 157286400, 210, {}, null, 600, null, null, f, null, null, t).\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'","app/services/application_settings/update_service.rb:51:in `update_settings'","app/services/application_settings/update_service.rb:12:in `execute'","ee/app/services/ee/application_settings/update_service.rb:25:in `execute'","lib/api/settings.rb:292:in `block in <class:Settings>'","ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'","lib/api/api_guard.rb:219:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/path_traversal_check.rb:25:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"exception.cause_class":"PG::NotNullViolation","exception.sql":"/*application:web,correlation_id:01HFNAPM6C586WKHQAVP1VRH2D,endpoint_id:PUT /api/:version/application/settings,db_config_name:main*/ UPDATE \"application_settings\" SET \"updated_at\" = $1, \"maintenance_mode\" = $2, \"maintenance_mode_message\" = $3 WHERE \"application_settings\".\"id\" = $4","queue_duration_s":0.044735,"redis_calls":21,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.005244,"redis_read_bytes":2971,"redis_write_bytes":2334,"redis_cache_calls":1,"redis_cache_duration_s":0.00036,"redis_cache_read_bytes":114,"redis_cache_write_bytes":96,"redis_feature_flag_calls":13,"redis_feature_flag_duration_s":0.002941,"redis_feature_flag_read_bytes":2677,"redis_feature_flag_write_bytes":894,"redis_sessions_calls":3,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.001025,"redis_sessions_read_bytes":180,"redis_sessions_write_bytes":1081,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.000918,"redis_shared_state_write_bytes":263,"db_count":7,"db_write_count":1,"db_cached_count":1,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.015,"db_main_duration_s":0.015,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.11755,"mem_objects":49803,"mem_bytes":6038495,"mem_mallocs":15124,"mem_total_bytes":8030615,"pid":3828,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01HFNAPM6C586WKHQAVP1VRH2D","meta.caller_id":"PUT /api/:version/application/settings","meta.remote_ip":"113.32.144.145","meta.feature_category":"not_owned","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","content_length":"61","request_urgency":"default","target_duration_s":1}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

\\\[ec2-user@ip-172-31-60-75 \\\~\\\]$ sudo gitlab-rake gitlab:env:infoSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.19 Bundler Version:2.4.20 Rake Version: 13.0.6 Redis Version: 7.0.13 Sidekiq Version:6.5.7 Go Version: unknownGitLab information Version: 16.5.1-ee Revision: 55da9ccb652 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://\ HTTP Clone URL: https://\/some-group/some-project.git SSH Clone URL: git@\:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers:GitLab Shell Version: 14.29.0 Repository storages: \\- default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shellGitaly \\- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket \\- default Version: 16.5.1 \\- default Git Version: 2.42.0

Results of GitLab application Check

Expand for output related to the GitLab application check 

\\\[ec2-user@ip-172-31-60-75 \\\~\\\]$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...GitLab Shell: ... GitLab Shell version \\\>= 14.29.0 ? ... OK (14.29.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successfulChecking GitLab Shell ... FinishedChecking Gitaly ...Gitaly: ... default ... OKChecking Gitaly ... FinishedChecking Sidekiq ...Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1Checking Sidekiq ... FinishedChecking Incoming Email ...Incoming Email: ... Reply by email is disabled in config/gitlab.ymlChecking Incoming Email ... FinishedChecking LDAP ...LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 4 users of 100 limit.Checking LDAP ... FinishedChecking GitLab App ...Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 11/4 ... yes 9/5 ... yes 9/6 ... yes 8/7 ... yes 6/8 ... yes 7/9 ... yes 11/11 ... yes 11/13 ... yes 11/14 ... yes 11/19 ... yes 22/21 ... yes 23/22 ... yes 41/33 ... yes 41/34 ... yes 41/35 ... yes 41/36 ... yes 41/37 ... yes 41/38 ... yes 82/53 ... yes 82/54 ... yes 82/55 ... yes 82/56 ... yes 82/57 ... yes 82/58 ... yes 82/59 ... yes 82/60 ... yes 82/61 ... yes 82/62 ... yes 93/63 ... yes 93/64 ... yes 94/65 ... yes 94/66 ... yes 102/67 ... yes 102/68 ... yes 102/69 ... yes 102/70 ... yes 102/71 ... yes 102/72 ... yes 102/73 ... yes 102/74 ... yes 102/75 ... yes 102/76 ... yes 102/77 ... yes 102/78 ... yes 116/79 ... yes 117/80 ... yes 117/81 ... yes 117/82 ... yes 82/83 ... yes 102/84 ... yes 136/86 ... yes Redis version \\\>= 6.0.0? ... yes Ruby version \\\>= 3.0.6 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 6 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)Checking GitLab App ... FinishedChecking GitLab subtasks ... Finished\\\[ec2-user@ip-172-31-60-75 \\\~\\\]$

Possible fixes

Edited by Kosuke Kamiya