storage: Move log entry pruning to a concurrent task in log manager
For #6529 (closed)
This MR transfers log entry pruning from TransactionManager to a concurrent goroutine managed by log.Manager. Previously, pruning was performed synchronously in the transaction processing loop, leading to head-of-line blocking. By relocating pruning to a dedicated goroutine, we reduce contention and enhance transaction throughput.
Log entry pruning is now moved to log.Manager. It is initiated whenever there is an update in the log consuming position. The pruning task operates independently. It ensures that only one pruning operation is active at a time. Errors encountered during pruning are communicated through the notify queue, and the Close method now waits for all active pruning tasks to finish before shutting down.
In this way, the entire lifecycle of log management is now encapsulated in log.Manager. Additionally, we no longer have a goroutine idling when it’s not needed.