Geo: Add Geo-secondary-specific routing in Workhorse
What does this MR do?
This MR is initially intended for showing something working end-to-end. Smaller MRs will be broken out for review and merge.
After the child MRs have been merged, this MR can retarget master. And this MR will just make Workhorse use Geo-secondary-specific routing if geo_proxy_url
is present in the /api/v4/geo/proxy
response.
the epic?
Where does this fit inDescription | Issue | MR(s) |
---|---|---|
Add endpoint /api/v4/geo/proxy
|
#324664 (closed) | !63343 (merged) |
Determine if this is a Geo Proxy by calling /api/v4/geo/proxy
|
#329670 (closed) | !60769 (merged) |
If this is a Geo Proxy, then use the Geo Proxy routing table | #329672 (closed) | !63321 (merged) You are here |
Cache the result of the request | #329671 (closed) | !66491 (merged) |
[Feature flag] Rollout of geo_secondary_proxy and GEO_SECONDARY_PROXY
|
#325732 (closed) |
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
I have included a changelog entry, or it's not needed. (Does this MR need a changelog?) => Not needed. The API endpoint is feature flagged, and the major workhorse changes are behind environment variable as well. -
I have added/updated documentation, or it's not needed. (Is documentation required?) -
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
To do
-
404s on requests to webpack *.chunk.js - In GDK, this worked after reconfiguring
webpack.incremental
tofalse
on both primary and secondary. But in my browser ongitlab.com
, various*.chunk.js
files are fetched fromgitlab.com
as well asassets.gitlab-static.net
. - This doesn't happen when
/assets
requests are proxied.
- In GDK, this worked after reconfiguring
-
Browser console error Content Security Policy: The page’s settings blocked the loading of a resource at http://gdk.test:3809/sockjs-node/info?t=1622837693220 (“connect-src”).
- This doesn't happen when
/assets
requests are proxied. Because then the resource is athttp://gdk.test:3808
which is included in theContent-Security-Policy
response headers generated by the primary. - Idea for later: Somehow add secondary webpack addresses in
::Gitlab::ContentSecurityPolicy::ConfigLoader.allow_webpack_dev_server
.- I don't see this
sockjs-node
request from my browser ongitlab.com
.-
Is this used in production?
-
- I don't see this
- This doesn't happen when
-
400 Browser console error Firefox can’t establish a connection to the server at ws://gdk.test:3001/-/cable.
fromcommons-pages.groups.epics.index-pages.groups.epics.new-pages.groups.epics.show-pages.groups.iterati-8ee3690e.chunk.js:5839:25
=> #333579 (closed)- Without this MR on geo.staging.gitlab.com,
/-/cable
404s. - Without this MR locall, without GEO_SECONDARY_PROXY=1,
/-/cable
101s which I guess is good.
- Without this MR on geo.staging.gitlab.com,
-
502 requests. Workhorse error output badgateway: failed to receive response: dial unix /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket: socket: too many open files
. It goes away after restarting gitlab-workhorse then comes back.- I thought maybe this was because my WIP code was instantiating a new handler to proxy to the primary for every proxied request. I fixed that but this still happens.
- I changed the way I close the API call response body and I didn't see this again after browsing around for 5 minutes
-
How can I set the environment variable GEO_SECONDARY_PROXY=1
forgdk start gitlab-workhorse
? At the moment I rungitlab/workhorse/gitlab-workhorse
manually just so I can set this variable.
How to test this change locally on MacOS
- Change directory to your Geo secondary GDK (This feature should have no effect on a Geo primary or non-Geo instance, though even in those cases, Workhorse will begin to make new internal API calls):
cd /path/to/your/gdk-geo
- Build workhorse:
make gitlab-workhorse-update
After Workhorse is built, then you need to run Workhorse with an environment variable set GEO_SECONDARY_PROXY=1
. I don't know why this is so hard, but you have two options:
Option 1: Set GEO_SECONDARY_PROXY=1 in Procfile
- Edit your Geo GDK's
Procfile
- Add
GEO_SECONDARY_PROXY=1
to the Workhorse linegitlab-workhorse: exec /usr/bin/env
so it looks likegitlab-workhorse: exec /usr/bin/env GEO_SECONDARY_PROXY=1
(you only have to do this once until the next reconfigure) gdk restart gitlab-workhorse
Option 2: Run gitlab-workhorse bin manually so it can see your shell's environment variables
- Note the command that GDK runs to run workhorse:
ps aux | grep gitlab-workhorse
- Stop GDK-managed workhorse since you need to run it with an environment variable, which sounds like a pain to set system-wide on MacOS:
gdk stop gitlab-workhorse
- Set the environment variable and run Workhorse manually. This is what I run:
GEO_SECONDARY_PROXY=1 ./gitlab/workhorse/gitlab-workhorse -authSocket /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket -cableSocket /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket -listenAddr gdk.test:3001 -documentRoot /Users/mkozonogitlab/Developer/gdk-geo/gitlab/public -developmentMode -secretPath /Users/mkozonogitlab/Developer/gdk-geo/gitlab/.gitlab_workhorse_secret -config /Users/mkozonogitlab/Developer/gdk-geo/gitlab/workhorse/config.toml -logFormat json
When running Workhorse with the environment variable set, you should see Workhorse log output like Geo Proxy: Handle this request locally
or Geo Proxy: Forward this request
.
Edited by Michael Kozono