cache: Fix flaky cache walker test
Similar to 5c596203 (cache: Fix flaky cache walker test, 2021-04-28),
we have another race in our cache walker test which is flaky because we
kick off the async walker when building the configuration with
testcfg.Build()
even though we don't intend to.
Fix this race by disabling this functionality early on.