Implement Acquisition.WithContext
With the returned acquisition, it should be possible to wrap a context that is canceled when the acquisition is relinquished.
This would occur in the following scenarios:
- The acquisition is released
- The slot becomes unavailable/stomped
- The instance is removed
This will allow us to cancel tasks that are now impossible to complete immediately, rather than have them eventually timeout.
For example:
- Performing an SSH connection against an instance that was for some reason deleted (could happen for some reason by us or the cloud provider) will continue to try until it eventually times out.
- An existing connection that was performing work suddenly stalls and we rely on the keepalive timing out
For Runner, we'd effectively wrap the executor context with this, so that we stop doing everything if there's no way to execute a job anymore.
Optional: maybe we can use Go 1.21's new WithCancelCause()
internally so that we can bubble up the reason.
Usage would look something like this:
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
acq, err := ts.Acquire(ctx, "key")
...
defer ts.Release("key")
ctx, cancel := acq.WithContext(ctx)
defer cancel(nil)
info, err := acq.InstanceConnectInfo(ctx)
connector.Run(ctx, info, ConnectorOptions{}) // run script or w/e, immediately canceled when we know acq is terminated
Edited by Arran Walker