Flaky tests: Fix TestAdaptiveCalculator flaky tests (second take)
This is a revert of a revert (original MR). In that MR, we fixed a flaky test but it was reverted accidentally. This MR brings it back.
In TestAdaptiveCalculator, we set the calibration duration to 10 milliseconds. This test setup uses a manual ticker. This calibration duration is irrelevant to the actual calibration cycle of the calculation. That value was supposed to be a placeholder.
Unfortunately, the calculator uses this value to determine a timeout when polling events from the watchers. Although event polling is supposed to be instant, it might not be the case for slow machines. Thus, on CIs, even polling might return deadline exceeded unintentionally. This commit fixes this flake by setting the calibration duration to an unrealistically high value.
This flake can be reproduced in the local environment by adding a small delay to the implementation and hammer the test for enough number of times.
diff --git a/internal/limiter/adaptive_calculator.go b/internal/limiter/adaptive_calculator.go
index 793fc853c..d3f7f7fcf 100644
--- a/internal/limiter/adaptive_calculator.go
+++ b/internal/limiter/adaptive_calculator.go
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"math"
+ "math/rand"
"sync"
"sync/atomic"
"time"
@@ -196,6 +197,8 @@ func (c *AdaptiveCalculator) pollBackoffEvent(ctx context.Context) {
ctx, cancel := context.WithTimeout(ctx, c.calibration)
defer cancel()
+ time.Sleep(time.Duration(rand.Intn(11)) * time.Millisecond)
+
for _, w := range c.watchers {
// If the context is cancelled, return early.
if ctx.Err() != nil {
diff --git a/internal/limiter/adaptive_calculator_test.go b/internal/limiter/adaptive_calculator_test.go
index 5c2a5dae7..d4965bdf4 100644
--- a/internal/limiter/adaptive_calculator_test.go
+++ b/internal/limiter/adaptive_calculator_test.go
@@ -55,8 +55,6 @@ func TestAdaptiveCalculator_realTimerTicker(t *testing.T) {
func TestAdaptiveCalculator(t *testing.T) {
t.Parallel()
- testhelper.SkipQuarantinedTest(t, "https://gitlab.com/gitlab-org/gitaly/-/issues/5467")
-
tests := []struct {
desc string
limits []AdaptiveLimiter
Before
go test -race -failfast --count 100 ./internal/limiter/ -run TestAdaptiveCalculator
--- FAIL: TestAdaptiveCalculator (0.28s)
--- FAIL: TestAdaptiveCalculator/Additive_increase_multiple_limits_until_a_backoff_event (0.05s)
adaptive_calculator_test.go:575:
Error Trace: /Users/qmnguyen/www/gitlab-development-kit/gitaly/internal/limiter/adaptive_calculator_test.go:575
Error: Not equal:
expected: []int{25, 26, 27, 28, 29, 30, 15, 16}
actual : []int{25, 26, 27, 28, 29, 30, 31, 15}
Diff:
--- Expected
+++ Actual
@@ -7,4 +7,4 @@
(int) 30,
- (int) 15,
- (int) 16
+ (int) 31,
+ (int) 15
}
Test: TestAdaptiveCalculator/Additive_increase_multiple_limits_until_a_backoff_event
FAIL
FAIL gitlab.com/gitlab-org/gitaly/v16/internal/limiter 0.879s
After
go test -race -failfast --count 100 ./internal/limiter/ -run TestAdaptiveCalculator
ok gitlab.com/gitlab-org/gitaly/v16/internal/limiter 63.266s