Move cURL output to the job output
Closes #355731 (closed)
Background
Frustration coming from the curl_output.txt artifact not being available when a job fails, which is when it's needed the most (see #355731 (closed) for details).
What was the problem?
curl -o a_file.txt only writes the output received from the server. It does not write cURL errors, for instance when a host is not accessible (i.e. when the review app is not yet ready). You can convince yourself locally by copy/pasting this script:
{
# Creates an output file on success
curl -o test-output-success.txt https://www.google.com
[ -f "test-output-success.txt" ] && echo 1 || echo 0
# Does not create an output file on failure
curl -o test-output-failure.txt https://www.google2.com
[ -f "test-output-failure.txt" ] && echo 1 || echo 0
}What this MR does
- We move the curl output to the job
- Keep the curl output not too verbose, but still informative, by just showing the failures thanks to the --fail flag
Results
CI_ENVIRONMENT_URL=https://httpstat.us/200
Verifying deployment at https://httpstat.us/200
Review app is deployed to https://httpstat.us/200CI_ENVIRONMENT_URL=https://httpstat.us/201
Verifying deployment at https://httpstat.us/201
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11 100 11 0 0 16 0 --:--:-- --:--:-- --:--:-- 17
Expected HTTP status 200: received 201
Retrying 2...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11 100 11 0 0 16 0 --:--:-- --:--:-- --:--:-- 17
Expected HTTP status 200: received 201
Retrying 1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11 100 11 0 0 16 0 --:--:-- --:--:-- --:--:-- 17
Expected HTTP status 200: received 201
Review app is not available at https://httpstat.us/201: see the logs from cURL above for more detailsCI_ENVIRONMENT_URL=https://httpstat.us/302
Verifying deployment at https://httpstat.us/302
Review app is deployed to https://httpstat.us/302Importantly, I would expect a redirect on review apps (due to the ingress controller, but I might be wrong on that). I suspect it's the main reason why we have the -L flag in the curl command...
As long as we end up on a 200 after the redirects, the script will be happy (which is what's expected in my view).
CI_ENVIRONMENT_URL=https://httpstat.us/404
Verifying deployment at https://httpstat.us/404
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
Expected HTTP status 200: received 404
Retrying 2...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
Expected HTTP status 200: received 404
Retrying 1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
Expected HTTP status 200: received 404
Review app is not available at https://httpstat.us/404: see the logs from cURL above for more detailsCI_ENVIRONMENT_URL=https://www.google2.com
Verifying deployment at https://www.google2.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: www.google2.com
Expected HTTP status 200: received 000
Retrying 2...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: www.google2.com
Expected HTTP status 200: received 000
Retrying 1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: www.google2.com
Expected HTTP status 200: received 000
Review app is not available at https://www.google2.com: see the logs from cURL above for more detailsSteps to reproduce locally
docker run -it registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3.5-kubectl1.17 bashFrom inside the container, copy/paste the following snippet:
{
CI_ENVIRONMENT_URL=https://httpstat.us/200
function echoerr() {
local header="${2:-no}"
if [ "${header}" != "no" ]; then
printf "\n\033[0;31m** %s **\n\033[0m" "${1}" >&2;
else
printf "\033[0;31m%s\n\033[0m" "${1}" >&2;
fi
}
function echoinfo() {
local header="${2:-no}"
if [ "${header}" != "no" ]; then
printf "\n\033[0;33m** %s **\n\033[0m" "${1}" >&2;
else
printf "\033[0;33m%s\n\033[0m" "${1}" >&2;
fi
}
function retry() {
if eval "$@"; then
return 0
fi
for i in 2 1; do
sleep 3s
echo "Retrying $i..."
if eval "$@"; then
return 0
fi
done
return 1
}
function test_url() {
local url="${1}"
curl --fail --output /dev/null "${url}"
}
function verify_deploy() {
echoinfo "Verifying deployment at ${CI_ENVIRONMENT_URL}"
if retry "test_url \"${CI_ENVIRONMENT_URL}\""; then
echoinfo "Review app is deployed to ${CI_ENVIRONMENT_URL}"
return 0
else
echoerr "Review app is not available at ${CI_ENVIRONMENT_URL}: see the logs from cURL above for more details"
return 1
fi
}
# Main script
verify_deploy
}Edited by David Dieulivol