Skip to content

Mattermost 502 bad gateway after upgrade from 9.5 to 10.0

Summary

Upgraded from 9.5 to 10.0 using Omnibus on Debian 7. Mattermost shows 502 bad gateway nginx error.

Steps to reproduce

apt-get update && apt-get install gitlab-ce

What is the current bug behavior?

Mattermost shows 502 bad gateway nginx error.

What is the expected correct behavior?

Mattermost works

Relevant logs and/or screenshots

last error of /var/log/gitlab/nginx/gitlab_matermost_error.log

2017/09/22 16:31:47 [error] 23035#0: *14 connect() failed (111: Connection refused) while connecting to upstream, client: 10.34.4.22, server: mattermost.mydomain.ca, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "mattermost.mydomain.ca"

errors in /var/log/gitlab/mattermost/current

2017-09-22_20:43:36.34773 github.com/mattermost/platform/utils.configureLog(0xc422477820)
2017-09-22_20:43:36.34774       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:165 +0x23f
2017-09-22_20:43:36.34777 github.com/mattermost/platform/utils.LoadConfig(0x7ffed3884ee5, 0x26)
2017-09-22_20:43:36.34779       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:380 +0x427
2017-09-22_20:43:36.34781 github.com/mattermost/platform/utils.InitAndLoadConfig(0x7ffed3884ee5, 0x26, 0x7f0d69ce53a8, 0x110d51f)
2017-09-22_20:43:36.34784       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:278 +0x61
2017-09-22_20:43:36.34787 main.runServer(0x7ffed3884ee5, 0x26)
2017-09-22_20:43:36.34789       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:47 +0x50
2017-09-22_20:43:36.34791 main.runServerCmd(0x1733160, 0xc4201ac6a0, 0x0, 0x2, 0x0, 0x0)
2017-09-22_20:43:36.34796       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:42 +0xe0
2017-09-22_20:43:36.34798 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).execute(0x1733160, 0xc420010160, 0x2, 0x2, 0x1733160, 0xc420010160)
2017-09-22_20:43:36.34802       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:649 +0x457
2017-09-22_20:43:36.34805 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1733160, 0x1735e60, 0x0, 0x0)
2017-09-22_20:43:36.34808       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:728 +0x339
2017-09-22_20:43:36.34810 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).Execute(0x1733160, 0xc4200001a0, 0xc4200001a0)
2017-09-22_20:43:36.34813       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:687 +0x2b
2017-09-22_20:43:36.34815 main.main()
2017-09-22_20:43:36.34816       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/mattermost.go:33 +0x2d
2017-09-22_20:43:37.58021 FileLogWriter("/var/log/gitlab/mattermost/mattermost.log/mattermost.log"): open /var/log/gitlab/mattermost/mattermost.log/mattermost.log: not a directory
2017-09-22_20:43:37.58241 panic: runtime error: invalid memory address or nil pointer dereference
2017-09-22_20:43:37.58243 [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x95da6f]
2017-09-22_20:43:37.58246
2017-09-22_20:43:37.58246 goroutine 1 [running]:
2017-09-22_20:43:37.58249 github.com/mattermost/platform/utils.configureLog(0xc4223e9820)
2017-09-22_20:43:37.58250       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:165 +0x23f
2017-09-22_20:43:37.58253 github.com/mattermost/platform/utils.LoadConfig(0x7ffe20235ee5, 0x26)
2017-09-22_20:43:37.58255       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:380 +0x427
2017-09-22_20:43:37.58257 github.com/mattermost/platform/utils.InitAndLoadConfig(0x7ffe20235ee5, 0x26, 0x7fc8a4e85550, 0x41189f)
2017-09-22_20:43:37.58260       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:278 +0x61
2017-09-22_20:43:37.58263 main.runServer(0x7ffe20235ee5, 0x26)
2017-09-22_20:43:37.58264       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:47 +0x50
2017-09-22_20:43:37.58267 main.runServerCmd(0x1733160, 0xc4201acbe0, 0x0, 0x2, 0x0, 0x0)
2017-09-22_20:43:37.58271       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:42 +0xe0
2017-09-22_20:43:37.58273 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).execute(0x1733160, 0xc420010160, 0x2, 0x2, 0x1733160, 0xc420010160)
2017-09-22_20:43:37.58278       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:649 +0x457
2017-09-22_20:43:37.58280 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1733160, 0x1735e60, 0x0, 0xc42012bf78)
2017-09-22_20:43:37.58283       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:728 +0x339
2017-09-22_20:43:37.58289 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).Execute(0x1733160, 0xc4200001a0, 0xc4200001a0)
2017-09-22_20:43:37.58292       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:687 +0x2b
2017-09-22_20:43:37.58295 main.main()
2017-09-22_20:43:37.58296       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/mattermost.go:33 +0x2d
2017-09-22_20:43:38.78248 FileLogWriter("/var/log/gitlab/mattermost/mattermost.log/mattermost.log"): open /var/log/gitlab/mattermost/mattermost.log/mattermost.log: not a directory
2017-09-22_20:43:38.78468 panic: runtime error: invalid memory address or nil pointer dereference
2017-09-22_20:43:38.78470 [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x95da6f]
2017-09-22_20:43:38.78473
2017-09-22_20:43:38.78474 goroutine 1 [running]:
2017-09-22_20:43:38.78476 github.com/mattermost/platform/utils.configureLog(0xc4222a1820)
2017-09-22_20:43:38.78477       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:165 +0x23f
2017-09-22_20:43:38.78480 github.com/mattermost/platform/utils.LoadConfig(0x7ffceb9eaee5, 0x26)
2017-09-22_20:43:38.78482       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:380 +0x427
2017-09-22_20:43:38.78484 github.com/mattermost/platform/utils.InitAndLoadConfig(0x7ffceb9eaee5, 0x26, 0x7fde51baa3a8, 0x110d51f)
2017-09-22_20:43:38.78487       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:278 +0x61
2017-09-22_20:43:38.78489 main.runServer(0x7ffceb9eaee5, 0x26)
2017-09-22_20:43:38.78491       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:47 +0x50
2017-09-22_20:43:38.78493 main.runServerCmd(0x1733160, 0xc4201ac6e0, 0x0, 0x2, 0x0, 0x0)
2017-09-22_20:43:38.78497       /var/lib/jenkins/jobs/msr/jobs/d/jobs/platform-release/workspace/src/github.com/mattermost/platform/cmd/platform/server.go:42 +0xe0

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 7.9
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.0.0
Redis Version:  3.2.5
Git Version:    2.13.5
Sidekiq Version:5.0.4
Go Version:     unknown


GitLab information
Version:        10.0.0
Revision:       bbf5c73
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.mydomain.ca
HTTP Clone URL: https://gitlab.mydomain.ca/some-group/some-project.git
SSH Clone URL:  git@gitlab.mydomain.ca:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        5.9.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Edited by Patrick Wasp