Skip to content

GitLab Migration: follow redirects when doing network requests with `BulkImports::Http`

George Koltsov requested to merge georgekoltsov/update-bulk-imports-http-client-to-follow-redirects into master

What does this MR do and why?

This MR fixes an issue with GitLab Migration (aka BulkImports https://docs.gitlab.com/ee/api/import.html) when exported relations are stored in AWS(ish) objects storage. As observed in #368091 (comment 1029271800) whenever destination GitLab instance is trying to download exported relation from source GitLab instance - it 'fails' by replying with 303 See Other which is a URL where exported relation can be found (AWS S3 bucket URL) and we don't follow redirects when using GitLab::HTTP.

(byebug) response.request
#<HTTParty::Request:0x00007fac7d6678b8 @changed_hosts=false, @credentials_sent=false, @http_method=Net::HTTP::Head, @options={:limit=>5, :assume_utf16_is_big_endian=>true, :default_params=>{}, :follow_redirects=>false, :parser=>HTTParty::Parser, :uri_adapter=>URI, :connection_adapter=>Gitlab::HTTPConnectionAdapter, :connection_adapter_options=>nil, :open_timeout=>10, :read_timeout=>20, :write_timeout=>30, :headers=>{"Content-Type"=>"application/json", "Authorization"=>"Bearer ypCa3Dzb23o5nvsixwPA"}, :maintain_method_across_redirects=>true}, @path=#<URI::HTTP http://gdk.test:3000/api/v4/groups/mygroup/export_relations/download?relation=milestones>, @last_uri=#<URI::HTTP http://gdk.test:3000/api/v4/groups/mygroup/export_relations/download?relation=milestones>, @raw_request=#<Net::HTTP::Head HEAD>, @last_response=#<Net::HTTPSeeOther 303 See Other readbody=true>>


(byebug) response

#<HTTParty::Response:0xf73f0 parsed_response=nil, @response=#<Net::HTTPSeeOther 303 See Other readbody=true>, @headers={"server"=>["nginx/1.21.3"], "date"=>["Mon, 18 Jul 2022 10:32:00 GMT"], "content-type"=>["text/plain"], "connection"=>["close"], "cache-control"=>["no-cache"], "location"=>["http://127.0.0.1:9000/uploads/bulk_imports/export_upload/export_file/343/milestones.ndjson.gz?X-Amz-Expires=600&X-Amz-Date=20220718T103200Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220718%2Fgdk%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=b493cbf3f282f0d3bc7a389db6f88c38d7355df5fadccb4485a215588885b000"], "vary"=>["Origin"], "x-content-type-options"=>["nosniff"], "x-frame-options"=>["SAMEORIGIN"], "x-request-id"=>["01G88DTRW7HVHF74ZJ0CTT80ZP"], "x-runtime"=>["0.167333"]}>

This only happens for AWS (and similar object storages, like Minio), where a request to download file redirects to another URL. File stored in Google Cloud Storage server the file directly. Locally stored files are also served without redirection.

This MR updates our BulkImports::Http client to follow redirections (with limit 2) to allow download of exported relations from AWS object storage.

Mentions #368091 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports