"Bad Gateway" using Pages for a merge request artifacts on Omnibus
Summary
When using the expose_as:
feature for Pages with a file that is large enough to trigger chunked delivery, nginx returns Bad Gateway for the pages URL due to an invalid Content-Length header.
Steps to reproduce
-
docker run -itp 80:80 --shm-size=256m --name=gitlab gitlab/gitlab-ce
-
docker exec -it gitlab bash
to get a shell, in which you canapt update && apt install vim
to get an editor. -
Inside the docker exec shell,
vim /etc/gitlab/gitlab.rb
and make the following settings changes:-
external_url
needs to be set correctly for your environment, because the new "runner registration flow" in 16.x will attempt to redirect you to whatever it thinks its hostname is. I usedtest-gitlab
. -
pages_external_url
set to a different domain name thanexternal_url
. I useddobby
, in loving tribute to what tracking this bug down felt like gitlab_pages['enable'] = true
-
-
restart the container with
docker restart gitlab
-
Adjust your local DNS or
/etc/hosts
such that whatever names you made up forexternal_url
andpages_external_url
all point at your local machine. You will need to add extra hosts entries forroot.<your_pages_external_url>
andprojects.<your_pages_external_url>
. With my particular URL settings, I added this hosts entry:127.0.0.1 test-gitlab dobby root.dobby projects.dobby
-
Register a gitlab runner. I used
docker run --rm -it gitlab-runner
for this. -
Make a test project which just ensures Gitlab Pages are working and enabled.
-
Try to make a merge request against that test project that has this code in
.gitlab-ci.yaml
(or just push my test project on .com up
pages for merge requests:
stage: build
image: python:3
script:
- mkdir public
- echo "<h1>Hello world</h1><pre>" > public/index.html
- python -c 'print(repr(list(range(3*1024))).encode("ascii", "xmlcharrefreplace").decode("ascii"))' >> public/index.html
- echo "</pre>" >> public/index.html
artifacts:
expose_as: "demonstration pages site"
paths:
- public/index.html
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- The merge request page will offer to show you some artifacts. Click this button to view the artifacts in Pages.
Example Project
Note that this bug does not reproduce on .com, however this project can be pushed up to a test omnibus instance to reproduce the issue.
What is the current bug behavior?
502 Bad Gateway
message from nginx
What is the expected correct behavior?
You should see an equivalent to this on your local instance.
Relevant logs and/or screenshots
Pages logs
{
"level": "info",
"msg": "http: invalid Content-Length of \"-1\"",
"time": "2023-06-30T16:10:06Z"
}
{
"content_type": "text/html; charset=utf-8",
"correlation_id": "01H46H38CYTRQNVN88XNRMZXB3",
"duration_ms": 123,
"host": "test-gitlab",
"level": "info",
"method": "GET",
"msg": "access",
"pages_https": true,
"proto": "HTTP/1.0",
"referrer": "http://test-gitlab/",
"remote_addr": "REDACTED",
"remote_ip": "REDACTED",
"status": 200,
"system": "http",
"time": "2023-06-30T16:10:06Z",
"ttfb_ms": 123,
"uri": "/-/expose_as-test-project/-/jobs/3/artifacts/public/index.html",
"user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
"written_bytes": 12158
}
The NGINX logs
2023/06/30 16:27:32 [error] 1725080#0: *695610 upstream sent invalid "Content-Length" header: "Content-Length: -1" while reading response header from upstream, client: REDACTED, server: ~^(?<group>.*)\.dobby$, request: "GET /-/expose_as-test-project/-/jobs/3/artifacts/public/index.html HTTP/2.0", upstream: "http://127.0.0.1:8090/-/expose_as-test-project/-/jobs/3/artifacts/public/index.html", host: "root.dobby"
The api_json.log entry
{
"time": "2023-06-30T21:35:52.695Z",
"severity": "INFO",
"duration_s": 0.14194,
"db_duration_s": 0.04602,
"view_duration_s": 0.09592,
"status": 200,
"method": "GET",
"path": "/api/v4/projects/root%2Fexpose_as-test-project/jobs/14/artifacts/public/index.html",
"params": [],
"host": "test-gitlab",
"remote_ip": "REDACTED, REDACTED, 127.0.0.1",
"ua": "Go-http-client/1.1",
"route": "/api/:version/projects/:id/jobs/:job_id/artifacts/*artifact_path",
"user_id": 1,
"username": "root",
"token_type": "OauthAccessToken",
"token_id": 3,
"queue_duration_s": 0.026851,
"redis_calls": 8,
"redis_duration_s": 0.002183,
"redis_read_bytes": 1418,
"redis_write_bytes": 670,
"redis_feature_flag_calls": 7,
"redis_feature_flag_duration_s": 0.001635,
"redis_feature_flag_read_bytes": 1418,
"redis_feature_flag_write_bytes": 564,
"redis_shared_state_calls": 1,
"redis_shared_state_duration_s": 0.000548,
"redis_shared_state_write_bytes": 106,
"db_count": 8,
"db_write_count": 0,
"db_cached_count": 0,
"db_replica_count": 0,
"db_primary_count": 8,
"db_main_count": 5,
"db_ci_count": 3,
"db_main_replica_count": 0,
"db_ci_replica_count": 0,
"db_replica_cached_count": 0,
"db_primary_cached_count": 0,
"db_main_cached_count": 0,
"db_ci_cached_count": 0,
"db_main_replica_cached_count": 0,
"db_ci_replica_cached_count": 0,
"db_replica_wal_count": 0,
"db_primary_wal_count": 0,
"db_main_wal_count": 0,
"db_ci_wal_count": 0,
"db_main_replica_wal_count": 0,
"db_ci_replica_wal_count": 0,
"db_replica_wal_cached_count": 0,
"db_primary_wal_cached_count": 0,
"db_main_wal_cached_count": 0,
"db_ci_wal_cached_count": 0,
"db_main_replica_wal_cached_count": 0,
"db_ci_replica_wal_cached_count": 0,
"db_replica_duration_s": 0,
"db_primary_duration_s": 0.011,
"db_main_duration_s": 0.008,
"db_ci_duration_s": 0.003,
"db_main_replica_duration_s": 0,
"db_ci_replica_duration_s": 0,
"cpu_s": 0.134418,
"mem_objects": 43611,
"mem_bytes": 11513995,
"mem_mallocs": 33403,
"mem_total_bytes": 13258435,
"pid": 15514,
"worker_id": "puma_16",
"rate_limiting_gates": [],
"correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
"meta.caller_id": "GET /api/:version/projects/:id/jobs/:job_id/artifacts/*artifact_path",
"meta.remote_ip": "172.17.0.2",
"meta.feature_category": "build_artifacts",
"meta.user": "root",
"meta.user_id": 1,
"meta.project": "root/expose_as-test-project",
"meta.root_namespace": "root",
"meta.client_id": "user/1",
"request_urgency": "low",
"target_duration_s": 5,
"response_bytes": 2
}
gitlab-workhorse
{
"archive": "/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2023_06_30/14/31/artifacts.zip",
"correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
"entry": "cHVibGljL2luZGV4Lmh0bWw=\n",
"level": "info",
"msg": "SendEntry: sending",
"path": "/api/v4/projects/root/expose_as-test-project/jobs/14/artifacts/public/index.html",
"time": "2023-06-30T21:35:52Z"
}
{
"content_type": "text/html; charset=utf-8",
"correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
"duration_ms": 196,
"host": "test-gitlab",
"level": "info",
"method": "GET",
"msg": "access",
"proto": "HTTP/1.1",
"referrer": "",
"remote_addr": "127.0.0.1:0",
"remote_ip": "127.0.0.1",
"route": "^/api/",
"status": 200,
"system": "http",
"time": "2023-06-30T21:35:52Z",
"ttfb_ms": 195,
"uri": "/api/v4/projects/root%2Fexpose_as-test-project/jobs/14/artifacts/public/index.html",
"user_agent": "Go-http-client/1.1",
"written_bytes": 17356
}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
docker exec -i gitlab gitlab-rake gitlab:env:info | tee >(xclip)
System information System: Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.13 Bundler Version:2.4.14 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.1.1 Revision: 9ce736bb2cd Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: http://test-gitlab HTTP Clone URL: http://test-gitlab/some-group/some-project.git SSH Clone URL: git@test-gitlab:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.23.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
docker exec -i gitlab gitlab-rake gitlab:check SANITIZE=true | tee >(xclip)
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 1/1 ... yes 1/2 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
I'll edit this post if I find any. My team spent the better part of today just figuring out this much.
The problem began when we upgraded our real instance to 16.1.1 yesterday (from 16.0.4).
Edit: Thanks to @stargo (see comments) we now have a workaround.
# /etc/gitlab/gitlab.rb
pages_nginx['custom_gitlab_server_config'] = 'proxy_http_version 1.1;'