Sporadic SSH connection refused

A small subset of our builds will fail to connect to the ECS container, seems it's not given enough time to finish starting up. It usually connects when running the build again. Is there a way to increase the wait time to connect or add a retry?

Our containers are based off of centos:7.

Preparing environment
00:00
INFO[2020-09-28T15:20:38-05:00] Starting fargate                              PID=7928 version="0.2.0 (933d940)"
INFO[2020-09-28T15:20:38-05:00] Executing the command                         PID=7928 command=run_exec stage=prepare_script
INFO[2020-09-28T15:20:38-05:00] Reading file content                          PID=7928 command=run_exec file=/tmp/custom-executor211415217/script427137518/script. stage=prepare_script
INFO[2020-09-28T15:20:38-05:00] Fetching task data from metadata storage      PID=7928 command=run_exec stage=prepare_script
INFO[2020-09-28T15:20:38-05:00] Executing script in the task container        PID=7928 command=run_exec stage=prepare_script taskARN="arn:aws:ecs:ca-central-1:*:task/*
ERRO[2020-09-28T15:20:38-05:00] Application execution failed                  PID=7928 error="executing the script on the remote host: executing script on container with IP \"<IP>\": connecting to server: connecting to server \"<IP>:22\" as user \"root\": dial tcp <IP>:22: connect: connection refused"
ERROR: Job failed (system failure): prepare environment: exit status 2. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information