CI jobs are failing during scale-out/scale-in activities
Hi,
It seems the runner manager is trying to assign jobs to an instance that is about to be terminated. Also randomly I have seen jobs fail as well during scale-in/scale-out time. I have attached the config and logs to this issue.
@ajwalker You were very helpful with understanding our environment last time so I trust you can help with getting us resolved with this as soon as possible.
best regards,
Kayode
Config
concurrent = 50 check_interval = 0 shutdown_timeout = 0[session_server] session_timeout = 1800[[runners]] name = "ip-10-2-68-203.shs.grn.XXXX.XXX.uk.localgrn.XXXX.XXX 10.2.68.203 172.17.0.1" url = "https://gitlab.XXXX.XXX.uk.local/" id = 60 token = "X-MDUoPyvVNaqvYg8GsL" token_obtained_at = 2023-11-23T17:46:49Z token_expires_at = 0001-01-01T00:00:00Z executor = "docker-autoscaler" environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"https://artifactory.XXXX.XXX.uk.local\":{\"auth\":\"c3ZjLXBmbS1hcnQtZ2xiLXJ1bjpBUDVEN1pyV1RzdUd4OURqOTIyWUFXV1NqNDRoYkU5NDNTaXJISA==\"}}}", "AWS_STS_REGIONAL_ENDPOINTS=regional", "http_proxy=http://proxy.shs.grn.XXXX.XXX.uk.local:3128", "https_proxy=http://proxy.shs.grn.XXXX.XXX.uk.local:3128", "no_proxy=sts.eu-west-2.amazonaws.com,169.254.169.254,localhost,gitlab.XXXX.XXX.uk.local,gitlab.grn.XXXX.XXX.uk.local,artifactory.XXXX.XXX.uk.local,*.grafana-workspace.eu-west-2.amazonaws.com,grafana.XXXX.XXX.local"] [runners.docker] tls_verify = false image = "artifactory.XXXX.XXX.uk.local/platform-dockerio/library/alpine:3.18.2" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache", "/etc/pki/ca-trust/source/anchors/red-ca-bundle.crt:/etc/gitlab-runner/certs/ca.crt:ro", "/home/gitlab-runner/.redterraformrc:/root/.terraformrc"] shm_size = 0 network_mtu = 0 [runners.autoscaler] capacity_per_instance = 5 max_use_count = 500 max_instances = 5 plugin = "fleeting-plugin-aws" [runners.autoscaler.plugin_config] name = "grnss-glbrunner-asg" [runners.autoscaler.connector_config] username = "ec2-user" password = "" key_path = "" use_static_credentials = false keepalive = "0s" timeout = "0s" use_external_addr = true[[runners.autoscaler.policy]] idle_count = 1 idle_time = "20m0s" scale_factor = 0.0 scale_factor_limit = 0[[runners]] name = "ip-10-2-68-203.shs.grn.XXXX.XXX.uk.localgrn.XXXX.XXX 10.2.68.203 172.17.0.1" url = "https://gitlab.grn.XXXX.XXX.uk.local/" id = 83 token = "KxrMkvpN1tNDniSsBeZD" token_obtained_at = 2023-11-23T17:46:50Z token_expires_at = 0001-01-01T00:00:00Z executor = "docker-autoscaler" environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"https://artifactory.grn.XXXX.XXX.uk.local\":{\"auth\":\"c3ZjLXBmbS1hcnQtZ2xiLXJ1bjpBUDhBOFdkWFJpbzhjYU54YlZMYWpVU0FFaVFaN3dUZlJwb3Fybw==\"}}}", "AWS_STS_REGIONAL_ENDPOINTS=regional", "http_proxy=http://proxy.shs.grn.XXXX.XXX.uk.local:3128", "https_proxy=http://proxy.shs.grn.XXXX.XXX.uk.local:3128", "no_proxy=sts.eu-west-2.amazonaws.com,169.254.169.254,localhost,gitlab.grn.XXXX.XXX.uk.local,artifactory.grn.XXXX.XXX.uk.local,*.grafana-workspace.eu-west-2.amazonaws.com,grafana.XXXX.XXX.local"] [runners.cache] MaxUploadedArchiveSize = 0 [runners.docker] tls_verify = false image = "artifactory.grn.XXXX.XXX.uk.local/platform-dockerio/library/alpine:3.18.2" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache", "/etc/ssl/certs/ca-bundle.crt:/etc/gitlab-runner/certs/ca.crt:ro", "/home/gitlab-runner/.terraformrc:/root/.terraformrc"] shm_size = 0 network_mtu = 0 [runners.autoscaler] capacity_per_instance = 5 max_use_count = 500 max_instances = 5 plugin = "fleeting-plugin-aws" [runners.autoscaler.plugin_config] name = "grnss-glbrunner-asg" [runners.autoscaler.connector_config] username = "ec2-user" password = "" key_path = "" use_static_credentials = false keepalive = "0s" timeout = "0s" use_external_addr = true[[runners.autoscaler.policy]] idle_count = 1 idle_time = "20m0s" scale_factor = 0.0 scale_factor_limit = 0
Logs
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.345Z [INFO] increasing instances: amount=1 group=aws/eu-west-2/grnss-glbrunner-asg
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.455Z [WARN] plugin: plugin configured with a nil SecureConfig
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.455Z [DEBUG] plugin: starting plugin: path=/usr/bin/fleeting-plugin-aws args=["fleeting-plugin-aws"]
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.455Z [DEBUG] plugin: plugin started: path=/usr/bin/fleeting-plugin-aws pid=18629
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.455Z [DEBUG] plugin: waiting for RPC address: path=/usr/bin/fleeting-plugin-aws
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.461Z [DEBUG] plugin.fleeting-plugin-aws: plugin address: address=/tmp/plugin1313109575 network=unix timestamp=2024-01-11T09:17:24.461Z
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.461Z [DEBUG] plugin: using plugin: version=0
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.462Z [TRACE] plugin.stdio: waiting for stdio data
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.491Z [INFO] increasing instances response: group=aws/eu-west-2/grnss-glbrunner-asg num_requested=1 num_successful=1
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.491Z [INFO] increase update: group=aws/eu-west-2/grnss-glbrunner-asg pending=1 total_pending=1 requesting=0
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.590Z [ERROR] plugin.fleeting-plugin-aws: new instances are not protected from scale in and should be: name=grnss-glbrunner-asg region=eu-west-2 timestamp=2024-01-11T09:17:24.590Z
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.590Z [INFO] plugin initialized: version=v0.4.0 build info="sha=d68db2e7; ref=main; go=go1.20.10; built_at=2023-11-07T18:59:52+0000; os_arch=linux/amd64"
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.693Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0977fbeac0b6f6f52 state=deleting cause=preexisted
Jan 11 09:17:24 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:24.693Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=creating cause=preexisted
Jan 11 09:17:25 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:25.599Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=creating cause=requested
Jan 11 09:17:36 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:36.504Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-028560d25cebe8d77 state=creating cause=unexpected
Jan 11 09:17:36 glbmanagergs101 gitlab-runner: 2024-01-11T09:17:36.900Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-028560d25cebe8d77 state=creating cause=unexpected
Jan 11 09:18:26 glbmanagergs101 dhclient[2034]: XMT: Solicit on eth0, interval 126200ms.
Jan 11 09:18:54 glbmanagergs101 gitlab-runner: 2024-01-11T09:18:54.794Z [INFO] instance pruned: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0977fbeac0b6f6f52 lifetime=1m31.614462048s
Jan 11 09:18:55 glbmanagergs101 gitlab-runner: 2024-01-11T09:18:55.513Z [INFO] instance pruned: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0977fbeac0b6f6f52 lifetime=1m30.820170962s
Jan 11 09:20:01 glbmanagergs101 systemd: Started Session 10050 of user root.
Jan 11 09:20:33 glbmanagergs101 dhclient[2034]: XMT: Solicit on eth0, interval 110200ms.
Jan 11 09:22:23 glbmanagergs101 dhclient[2034]: XMT: Solicit on eth0, interval 114770ms.
Jan 11 09:22:31 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:31.954Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=running
Jan 11 09:22:31 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:31.995Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=running
Jan 11 09:22:35 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:35.167Z [INFO] ready: instance=i-0bd23156121303050 took=3.213096301s
Jan 11 09:22:36 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:36.324Z [ERROR] ready up preparation failed: instance=i-0bd23156121303050 err="no data on pre-existing instance so removing for safety"
Jan 11 09:22:37 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:37.374Z [INFO] decreasing instances: amount=1 group=aws/eu-west-2/grnss-glbrunner-asg
Jan 11 09:22:37 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:37.662Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=deleting
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:38.454Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-0bd23156121303050 state=deleting
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:38.454Z [ERROR] plugin.fleeting-plugin-aws: out-of-sync capacity: region=eu-west-2 actual=1 expected=2 name=grnss-glbrunner-asg timestamp=2024-01-11T09:22:38.454Z
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161119 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=1 job#033[0;m=161119 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161119 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:38 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-0bd23156121303050 internal-address#033[0;m=10.2.68.197 job#033[0;m=161119 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161120 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=2 job#033[0;m=161120 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161120 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-0bd23156121303050 internal-address#033[0;m=10.2.68.197 job#033[0;m=161120 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161121 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=3 job#033[0;m=161121 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161121 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-0bd23156121303050 internal-address#033[0;m=10.2.68.197 job#033[0;m=161121 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161122 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=4 job#033[0;m=161122 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161122 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:39 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-0bd23156121303050 internal-address#033[0;m=10.2.68.197 job#033[0;m=161122 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161123 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=5 job#033[0;m=161123 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161123 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-0bd23156121303050 internal-address#033[0;m=10.2.68.197 job#033[0;m=161123 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:40.616Z [INFO] increasing instances: amount=1 group=aws/eu-west-2/grnss-glbrunner-asg
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:40.722Z [INFO] increasing instances response: group=aws/eu-west-2/grnss-glbrunner-asg num_requested=1 num_successful=1
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:40.722Z [INFO] increase update: group=aws/eu-west-2/grnss-glbrunner-asg pending=1 total_pending=1 requesting=0
Jan 11 09:22:40 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:40.962Z [ERROR] plugin.fleeting-plugin-aws: out-of-sync capacity: actual=2 expected=1 name=grnss-glbrunner-asg region=eu-west-2 timestamp=2024-01-11T09:22:40.962Z
Jan 11 09:22:41 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161119 job-log#033[0;m=0-476 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-475 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:22:42 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161120 job-log#033[0;m=0-476 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-475 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:22:42 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161121 job-log#033[0;m=0-476 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-475 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:22:42 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161122 job-log#033[0;m=0-476 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-475 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:22:43 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:43.174Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-028560d25cebe8d77 state=running
Jan 11 09:22:43 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161123 job-log#033[0;m=0-476 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-475 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:22:43 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:43.975Z [INFO] instance update: group=aws/eu-west-2/grnss-glbrunner-asg id=i-028560d25cebe8d77 state=running
Jan 11 09:22:44 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:44.188Z [INFO] ready: instance=i-028560d25cebe8d77 took=1.01351321s
Jan 11 09:22:44 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:44.252Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-05671620835ffd51b state=creating cause=unexpected
Jan 11 09:22:45 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:45.040Z [INFO] instance discovery: group=aws/eu-west-2/grnss-glbrunner-asg id=i-05671620835ffd51b state=creating cause=requested
Jan 11 09:22:49 glbmanagergs101 gitlab-runner: 2024-01-11T09:22:49.983Z [INFO] ready: instance=i-028560d25cebe8d77 took=6.007684419s
Jan 11 09:22:50 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161124 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:22:50 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=6 job#033[0;m=161124 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:22:50 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161124 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:22:50 glbmanagergs101 gitlab-runner: Dialing instance #033[0;m external-address#033[0;m= instance-id#033[0;m=i-028560d25cebe8d77 internal-address#033[0;m=10.2.68.202 job#033[0;m=161124 project#033[0;m=3263 runner#033[0;m=X-MDUoPy use-external-address#033[0;m=true
Jan 11 09:22:53 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161124 job-log#033[0;m=0-658 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-657 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:15 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=24.518820805 job#033[0;m=161124 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:15 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161124 job-log#033[0;m=0-3219 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=658-3218 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:15 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3219 checksum#033[0;m=crc32:b9105fdb job#033[0;m=161124 runner#033[0;m=X-MDUoPy
Jan 11 09:23:15 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3219 checksum#033[0;m=crc32:b9105fdb code#033[0;m=200 job#033[0;m=161124 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:15 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=5 job#033[0;m=161124 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:33 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=53.192490291 job#033[0;m=161122 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:33 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161122 job-log#033[0;m=0-3220 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=476-3219 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:33 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3220 checksum#033[0;m=crc32:e8f823c9 job#033[0;m=161122 runner#033[0;m=X-MDUoPy
Jan 11 09:23:33 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3220 checksum#033[0;m=crc32:e8f823c9 code#033[0;m=200 job#033[0;m=161122 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:33 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=4 job#033[0;m=161122 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:34 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=55.254405277 job#033[0;m=161120 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:34 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161120 job-log#033[0;m=0-3211 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=476-3210 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:34 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3211 checksum#033[0;m=crc32:769dcc63 job#033[0;m=161120 runner#033[0;m=X-MDUoPy
Jan 11 09:23:34 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3211 checksum#033[0;m=crc32:769dcc63 code#033[0;m=200 job#033[0;m=161120 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:34 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=3 job#033[0;m=161120 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=55.746892628 job#033[0;m=161123 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161123 job-log#033[0;m=0-3413 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=476-3412 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3413 checksum#033[0;m=crc32:d90adc96 job#033[0;m=161123 runner#033[0;m=X-MDUoPy
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3413 checksum#033[0;m=crc32:d90adc96 code#033[0;m=200 job#033[0;m=161123 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=2 job#033[0;m=161123 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=56.737781993 job#033[0;m=161121 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161121 job-log#033[0;m=0-3414 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=476-3413 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3414 checksum#033[0;m=crc32:2e74d6b8 job#033[0;m=161121 runner#033[0;m=X-MDUoPy
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3414 checksum#033[0;m=crc32:2e74d6b8 code#033[0;m=200 job#033[0;m=161121 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=1 job#033[0;m=161121 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Job succeeded #033[0;m duration_s#033[0;m=58.176333327 job#033[0;m=161119 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161119 job-log#033[0;m=0-3405 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=476-3404 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:36 glbmanagergs101 gitlab-runner: Updating job... #033[0;m bytesize#033[0;m=3405 checksum#033[0;m=crc32:34c4d5ba job#033[0;m=161119 runner#033[0;m=X-MDUoPy
Jan 11 09:23:37 glbmanagergs101 gitlab-runner: Submitting job to coordinator...ok #033[0;m bytesize#033[0;m=3405 checksum#033[0;m=crc32:34c4d5ba code#033[0;m=200 job#033[0;m=161119 job-status#033[0;m= runner#033[0;m=X-MDUoPy update-interval#033[0;m=0s
Jan 11 09:23:37 glbmanagergs101 gitlab-runner: Removed job from processing list #033[0;m builds#033[0;m=0 job#033[0;m=161119 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:38 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161125 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:38 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=1 job#033[0;m=161125 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:38 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:40 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161125 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:40 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: 11c8794b-f47b-4bfc-8647-8db692b7f00d, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161125 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:40 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:41 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161126 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:41 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=2 job#033[0;m=161126 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:41 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161126 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:41 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161125 job-log#033[0;m=0-963 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-962 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:43 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161126 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: 9c8ec68c-49f0-4860-a398-b3c3c7f3e080, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161126 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161126 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161126 job-log#033[0;m=0-963 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-962 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161127 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=3 job#033[0;m=161127 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:44 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161127 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161125 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: df74696c-4a3c-40f4-8eb1-60b0ab79cfeb, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161125 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161128 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=4 job#033[0;m=161128 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:46 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161128 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:47 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161126 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:47 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161127 job-log#033[0;m=0-427 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-426 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:49 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:49 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161128 job-log#033[0;m=0-427 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-426 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:50 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161127 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:50 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: dddc1db6-731a-4170-9477-bf6411303336, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161127 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:50 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161127 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:51 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161129 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:51 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=5 job#033[0;m=161129 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:51 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161129 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:52 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161126 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:52 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: 9f14149b-1735-4c6b-a2fb-36f367487eb3, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161126 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:52 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161126 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:53 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161127 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:54 glbmanagergs101 gitlab-runner: Appending trace to coordinator...ok #033[0;m code#033[0;m=202 job#033[0;m=161129 job-log#033[0;m=0-427 job-status#033[0;m=running runner#033[0;m=X-MDUoPy sent-log#033[0;m=0-426 status#033[0;m=202 Accepted update-interval#033[0;m=1m0s
Jan 11 09:23:54 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Failed to remove network for build #033[0;m #033[31;1merror#033[0;m=networksManager is undefined #033[31;1mjob#033[0;m=161125 #033[31;1mnetwork#033[0;m= #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:54 glbmanagergs101 gitlab-runner: #033[0;33mWARNING: Preparation failed: preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: f8b1689d-35e8-4367-b886-2bdf95dd47da, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[0;33mjob#033[0;m=161125 #033[0;33mproject#033[0;m=3263 #033[0;33mrunner#033[0;m=X-MDUoPy
Jan 11 09:23:54 glbmanagergs101 gitlab-runner: Will be retried in 3s ... #033[0;m job#033[0;m=161125 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:55 glbmanagergs101 gitlab-runner: Checking for jobs... received #033[0;m job#033[0;m=161130 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git runner#033[0;m=X-MDUoPy
Jan 11 09:23:55 glbmanagergs101 gitlab-runner: Added job to processing list #033[0;m builds#033[0;m=6 job#033[0;m=161130 max_builds#033[0;m=50 project#033[0;m=3263 repo_url#033[0;m=https://gitlab.XXXX.XXX.uk.local/XXXX-XXX/operational-monitoring-iac.git
Jan 11 09:23:55 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161130 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:55 glbmanagergs101 gitlab-runner: Preparing instance... #033[0;m job#033[0;m=161126 project#033[0;m=3263 runner#033[0;m=X-MDUoPy
Jan 11 09:23:57 glbmanagergs101 gitlab-runner: #033[31;1mERROR: Job failed (system failure): preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: f8b1689d-35e8-4367-b886-2bdf95dd47da, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance.#033[0;m #033[31;1mduration_s#033[0;m=19.321887358 #033[31;1mjob#033[0;m=161125 #033[31;1mproject#033[0;m=3263 #033[31;1mrunner#033[0;m=X-MDUoPyEdited by kayodeolajideHO