Error 502: http: panic serving 127.0.0.1:34402: unaligned 64-bit atomic operation

Summary

Error 502 on trying to get a gitlab pages

Steps to reproduce

Use Gitlab installation on PI (armhf/32bits version)
Create a project with template "Gitbook"
Run pipeline
Try open page

Example Project

May only done with arm32 installation. Using "GitBook template" without any edits.

What is the current bug behavior?

What is the expected correct behavior?

Get a page like this one: https://pages.gitlab.io/gitbook/ or https://ax10336.gitlab.io/gitbook_test

Relevant logs and/or screenshots

File: /var/log/gitlab/nginx/gitlab_pages_error.log

2021/09/08 15:45:32 [error] 19400#0: *150 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.123, server: ~^(?<group>.*)\.p\.somewhere\.nolan$, request: "GET /gitbook_test HTTP/2.0", upstream: "http://127.0.0.1:8090/gitbook_test", host: "sandbox.p.somewhere.nolan"
2021/09/08 15:45:32 [error] 19400#0: *150 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.123, server: ~^(?<group>.*)\.p\.somewhere\.nolan$, request: "GET /gitbook_test HTTP/2.0", upstream: "http://[::1]:8090/gitbook_test", host: "sandbox.p.somewhere.nolan"

File: /var/log/gitlab/gitlab-pages/current

{"content_type":"","correlation_id":"01FF2XY133G1SKZVQHYX09MWPH","duration_ms":0,"host":"sandbox.p.somewhere.nolan","level":"info","method":"GET","msg":"access","pages_host":"sandbox.p.somewhere.nolan","pages_https":true,"pages_project_id":136,"pages_project_prefix":"/gitbook_test/","pages_project_serving_type":"zip","proto":"HTTP/1.0","referrer":"","remote_addr":"192.168.1.123","remote_ip":"192.168.1.123","status":0,"system":"http","time":"2021-09-08T15:45:32+01:00","uri":"/gitbook_test","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":0}
2021/09/08 15:45:32 http: panic serving 127.0.0.1:34402: unaligned 64-bit atomic operation
goroutine 165 [running]:
net/http.(*conn).serve.func1(0x2c8e480)
        /usr/local/go/src/net/http/server.go:1824 +0x104
panic(0x6660a0, 0x815f30)
        /usr/local/go/src/runtime/panic.go:971 +0x4b4
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x305d3b4, 0x1, 0x0, 0x1f, 0x0)
        /usr/local/go/src/runtime/internal/atomic/asm_arm.s:233 +0x14
gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip.newArchive(0x305d380, 0xfc23ac00, 0x6, 0x0)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip/archive.go:72 +0x68
gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip.(*zipVFS).findOrCreateArchive(0x305d380, 0x825934, 0x2ee0108, 0x327c4d0, 0xa1, 0x0, 0x0, 0x0)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip/vfs.go:219 +0x180
gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip.(*zipVFS).findOrOpenArchive(0x305d380, 0x825934, 0x2ee0108, 0x327c4d0, 0xa1, 0x327c420, 0xa1, 0x570500, 0x3160230, 0x63e790)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip/vfs.go:242 +0x3c
gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip.(*zipVFS).Root(0x305d380, 0x825934, 0x2ee0108, 0x327c420, 0xa1, 0x0, 0x0, 0x56fc40, 0xbe6dec)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip/vfs.go:161 +0x90
gitlab.com/gitlab-org/gitlab-pages/internal/vfs.(*instrumentedVFS).Root(0x302dce8, 0x825934, 0x2ee0108, 0x327c420, 0xa1, 0x0, 0x0, 0xd, 0x0)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/vfs/vfs.go:38 +0x4c
gitlab.com/gitlab-org/gitlab-pages/internal/serving/disk.(*Reader).tryFile(0x2d83880, 0xa6bcc830, 0x2d86800, 0x30a2280, 0x315a2d0, 0x0, 0x0, 0xa1)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/serving/disk/reader.go:68 +0x60
gitlab.com/gitlab-org/gitlab-pages/internal/serving/disk.(*Disk).ServeFileHTTP(0x2d83880, 0xa6bcc830, 0x2d86800, 0x30a2280, 0x315a2d0, 0x0, 0x0, 0x0)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/serving/disk/serving.go:21 +0x2c
gitlab.com/gitlab-org/gitlab-pages/internal/serving.(*Request).ServeFileHTTP(...)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/serving/request.go:22
gitlab.com/gitlab-org/gitlab-pages/internal/domain.(*Domain).ServeFileHTTP(0x2d86780, 0xa6bcc830, 0x2d86800, 0x30a2280, 0x2d86780)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/domain/domain.go:139 +0x180
main.(*theApp).serveFileOrNotFoundHandler.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:273 +0x19c
net/http.HandlerFunc.ServeHTTP(0x325ba68, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/rs/cors.(*Cors).Handler.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/rs/cors@v1.7.0/cors.go:219 +0x170
net/http.HandlerFunc.ServeHTTP(0x324da00, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).accessControlMiddleware.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:261 +0xc8
net/http.HandlerFunc.ServeHTTP(0x324da10, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).auxiliaryMiddleware.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:244 +0x198
net/http.HandlerFunc.ServeHTTP(0x324da20, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).authMiddleware.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:228 +0x94
net/http.HandlerFunc.ServeHTTP(0x324da30, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).acmeMiddleware.func1(0xa6bcc830, 0x2d86800, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:217 +0xe0
net/http.HandlerFunc.ServeHTTP(0x324da40, 0xa6bcc830, 0x2d86800, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
gitlab.com/gitlab-org/labkit/log.AccessLogger.func1(0xa6bcc798, 0x315a180, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/gitlab.com/gitlab-org/labkit@v1.3.0/log/access_logger.go:23 +0xd0
net/http.HandlerFunc.ServeHTTP(0x324da70, 0xa6bcc798, 0x315a180, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0xa6bcc798, 0x315a150, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:100 +0xb0
net/http.HandlerFunc.ServeHTTP(0x317cb40, 0xa6bcc798, 0x315a150, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1(0xa6bcc798, 0x315a120, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:68 +0xc8
net/http.HandlerFunc.ServeHTTP(0x317cbe8, 0xa6bcc798, 0x315a120, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1(0xa6bcc798, 0x315a120, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:40 +0x98
net/http.HandlerFunc.ServeHTTP(0x317cc00, 0xa6bcc798, 0x315a120, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1(0xa6bcc798, 0x315a0f0, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:163 +0xb8
net/http.HandlerFunc.ServeHTTP(0x317cca8, 0xa6bcc798, 0x315a0f0, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0xa6bcc798, 0x315a0c0, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:196 +0xb8
net/http.HandlerFunc.ServeHTTP(0x317cd50, 0xa6bcc798, 0x315a0c0, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerTimeToWriteHeader.func1(0x8249d8, 0x2e0e0a0, 0x30a2280)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:137 +0x13c
net/http.HandlerFunc.ServeHTTP(0x317cdf8, 0x8249d8, 0x2e0e0a0, 0x30a2280)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).routingMiddleware.func1(0x8249d8, 0x2e0e0a0, 0x30a2100)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:174 +0xa0
net/http.HandlerFunc.ServeHTTP(0x324de60, 0x8249d8, 0x2e0e0a0, 0x30a2100)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).healthCheckMiddleware.func2(0x8249d8, 0x2e0e0a0, 0x30a2100)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:195 +0x64
net/http.HandlerFunc.ServeHTTP(0x317ce10, 0x8249d8, 0x2e0e0a0, 0x30a2100)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).customHeadersMiddleware.func1(0x8249d8, 0x2e0e0a0, 0x30a2100)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:204 +0x6c
net/http.HandlerFunc.ServeHTTP(0x324dea0, 0x8249d8, 0x2e0e0a0, 0x30a2100)
        /usr/local/go/src/net/http/server.go:2069 +0x34
gitlab.com/gitlab-org/labkit/correlation.InjectCorrelationID.func1(0x8249d8, 0x2e0e0a0, 0x30a2080)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/gitlab.com/gitlab-org/labkit@v1.3.0/correlation/inbound_http.go:35 +0x180
net/http.HandlerFunc.ServeHTTP(0x324deb0, 0x8249d8, 0x2e0e0a0, 0x30a2080)
        /usr/local/go/src/net/http/server.go:2069 +0x34
gitlab.com/gitlab-org/gitlab-pages/internal/rejectmethods.NewMiddleware.func1(0x8249d8, 0x2e0e0a0, 0x30a2080)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/internal/rejectmethods/middleware.go:25 +0x88
net/http.HandlerFunc.ServeHTTP(0x324dec0, 0x8249d8, 0x2e0e0a0, 0x30a2080)
        /usr/local/go/src/net/http/server.go:2069 +0x34
github.com/gorilla/handlers.ProxyHeaders.func1(0x8249d8, 0x2e0e0a0, 0x30a2080)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/.GOPATH/pkg/mod/github.com/gorilla/handlers@v1.4.2/proxy_headers.go:59 +0xec
net/http.HandlerFunc.ServeHTTP(0x324ded0, 0x8249d8, 0x2e0e0a0, 0x30a2080)
        /usr/local/go/src/net/http/server.go:2069 +0x34
main.(*theApp).proxyInitialMiddleware.func1(0x8249d8, 0x2e0e0a0, 0x30a2080)
        /var/cache/omnibus/src/gitlab-pages/src/gitlab.com/gitlab-org/gitlab-pages/app.go:305 +0x94
net/http.HandlerFunc.ServeHTTP(0x324dee0, 0x8249d8, 0x2e0e0a0, 0x30a2080)
        /usr/local/go/src/net/http/server.go:2069 +0x34
net/http.serverHandler.ServeHTTP(0x310f9e0, 0x8249d8, 0x2e0e0a0, 0x30a2080)
        /usr/local/go/src/net/http/server.go:2887 +0x88
net/http.(*conn).serve(0x2c8e480, 0x825934, 0x2da8020)
        /usr/local/go/src/net/http/server.go:1952 +0x7f0
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3013 +0x318

Output of checks

Gitlab instance info

# gitlab-rake gitlab:env:info

System information
System:         Raspbian 10
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.32.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.2.3
Revision:       2324101daeb
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://gitlab.somewhere.nolan
HTTP Clone URL: https://gitlab.somewhere.nolan/some-group/some-project.git
SSH Clone URL:  git@gitlab.somewhere.nolan:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        13.19.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

# LC_ALL=C apt-cache policy gitlab-ce | head
gitlab-ce:
  Installed: 14.2.3-ce.0
  Candidate: 14.2.3-ce.0
  Version table:
 *** 14.2.3-ce.0 500
        500 https://packages.gitlab.com/gitlab/raspberry-pi2/raspbian buster/main armhf Packages
        100 /var/lib/dpkg/status
     14.2.2-ce.0 500
        500 https://packages.gitlab.com/gitlab/raspberry-pi2/raspbian buster/main armhf Packages
     14.2.1-ce.0 500

Possible fixes

Maybe change something here: https://gitlab.com/gitlab-org/gitlab-pages/-/blob/85c4291724f713a37a047e9baacb9cd2187d9de2/internal/vfs/zip/archive.go#L72

~"devops::release" ~"group::release" Category:Pages

Edited Sep 08, 2021 by Æx