Skip to content
This repository was archived by the owner on Oct 13, 2023. It is now read-only.

Commit aae123a

Browse files
cpuguy83thaJeztah
authored andcommitted
Sleep before restarting event processing
This prevents restarting event processing in a tight loop. You can see this with the following steps: ```terminal $ containerd & $ dockerd --containerd=/run/containerd/containerd.sock & $ pkill -9 containerd ``` At this point you will be spammed with logs such as: ``` ERRO[2019-07-12T22:29:37.318761400Z] failed to get event error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused\"" module=libcontainerd namespace=plugins.moby ``` Without this change you can quickly end up with gigabytes of log data. Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit 1acaf2aabe000c6101501af321969df7c0ca5413) Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 3833f2a60bfe78f4a7966c862b736af0c36c3ede Component: engine
1 parent c7ae679 commit aae123a

1 file changed

Lines changed: 9 additions & 3 deletions

File tree

  • components/engine/libcontainerd/remote

components/engine/libcontainerd/remote/client.go

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -703,10 +703,16 @@ func (c *client) processEventStream(ctx context.Context, ns string) {
703703
errStatus, ok := status.FromError(err)
704704
if !ok || errStatus.Code() != codes.Canceled {
705705
c.logger.WithError(err).Error("failed to get event")
706-
go c.processEventStream(ctx, ns)
707-
} else {
708-
c.logger.WithError(ctx.Err()).Info("stopping event stream following graceful shutdown")
706+
707+
// rate limit
708+
select {
709+
case <-time.After(time.Second):
710+
go c.processEventStream(ctx, ns)
711+
return
712+
case <-ctx.Done():
713+
}
709714
}
715+
c.logger.WithError(ctx.Err()).Info("stopping event stream following graceful shutdown")
710716
}
711717
return
712718
case ev = <-eventStream:

0 commit comments

Comments
 (0)