feat(metrics): add operation counters, deletion duration, resume type label, and abnormal state detection by KeyOfSpectator · Pull Request #336 · openkruise/agents

KeyOfSpectator · 2026-05-06T08:30:56Z

What this PR does

Enhance Prometheus metrics across Controller and Manager layers with operation counters, deletion latency tracking, phase/condition mismatch detection, and cardinality-optimized label design to prevent time series explosion in production.

Changes

Label Cardinality Strategy

All Histogram and Counter metrics use only namespace as the dynamic label (no per-sandbox name) to prevent cardinality explosion. Only Gauge metrics that require per-sandbox granularity retain the name label.

Controller Layer — New Metrics

Metric	Type	Labels	Purpose
`sandbox_creation_total`	Counter	namespace, result	Track creation success/failure rate
`sandbox_pause_total`	Counter	namespace, result	Track pause operation completions
`sandbox_resume_total`	Counter	namespace, result	Track resume operation completions
`sandbox_deletion_duration_seconds`	Histogram	namespace	Measure deletion latency distribution
`sandbox_status_abnormal`	Gauge	namespace, name, type	Detect phase/condition stuck states

Controller Layer — Existing Metrics Enhanced

Metric	Change
`sandbox_creation_duration_seconds`	Histogram → HistogramVec (namespace only)
`sandbox_inplace_update_duration_seconds`	Histogram → HistogramVec (namespace only)
`sandbox_pause_duration_seconds`	Histogram → HistogramVec (namespace only)
`sandbox_resume_duration_seconds`	Histogram → HistogramVec (namespace only)

SandboxSet Layer

Metric	Type	Labels	Purpose
`sandboxset_sandboxes_created_total`	Counter	namespace	Track sandbox creation by set
`sandboxset_sandboxes_claimed_total`	Counter	namespace	Track consumption speed

SandboxClaim Layer

Metric	Type	Labels	Purpose
`sandboxset_claims_total`	Counter	namespace, result	Track claim operations
`sandboxclaim_expired_total`	Counter	namespace	Track TTL expiration deletions

Manager Layer

All metrics upgraded from plain Histogram/Counter to Vec with namespace label:

Duration histograms: sandbox_resume/delete/claim/clone_duration_seconds, sandbox_claim_retries
Counters: sandbox_claim_creation_responses, sandbox_claim_total, sandbox_clone_total, sandbox_pause/resume/delete_responses
Route sync: sandbox_route_sync_duration_seconds, sandbox_route_sync_total

E2B Layer

sandbox_snapshot_duration_seconds — Histogram with namespace label
sandbox_snapshot_total — Counter with namespace, result labels

Key Design Decisions

Cardinality optimization: Histogram and Counter use namespace-only to avoid O(n) series per sandbox. Gauge retains name for per-instance stuck detection.
Operation counters co-located with duration observations via extended recordConditionDuration() (optional counter parameter)
Deletion duration observed at cleanup time using time.Since(DeletionTimestamp)
Abnormal state detection covers pause_incomplete and resume_incomplete types
ConstLabel source distinguishes controller (k8s) vs manager (e2b) origin

How to verify

go test ./pkg/controller/sandbox/ ./pkg/controller/sandboxset/ ./pkg/controller/sandboxclaim/... ./pkg/sandbox-manager/ -count=1 -timeout 120s

kruise-bot · 2026-05-06T08:31:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign furykerry for approval by writing /assign @furykerry in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2026-05-06T08:35:18Z

Codecov Report

❌ Patch coverage is 98.86364% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 75.43%. Comparing base (edaad73) to head (6010f41).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/sandbox-manager/api.go	95.65%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #336      +/-   ##
==========================================
+ Coverage   74.65%   75.43%   +0.77%     
==========================================
  Files         141      142       +1     
  Lines        9836     9907      +71     
==========================================
+ Hits         7343     7473     +130     
+ Misses       2183     2127      -56     
+ Partials      310      307       -3

Flag	Coverage Δ
unittests	`75.43% <98.86%> (+0.77%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ation_seconds Remove high-cardinality labels (resume_reason, name) from the sandbox_resume_duration_seconds histogram to prevent label explosion. Only the namespace dimension is retained alongside the source=k8s const label. - Update metric definition to use []string{"namespace"} only - Simplify recording logic by removing resumeReason extraction - Remove deleteSandboxMetrics cleanup for this histogram since namespace-level metrics should not be deleted per-sandbox - Remove TestSandboxResumeDurationWithReason and related helpers - Update TestSandboxResumeDuration to match new label signature

…prevent cardinality explosion Remove the high-cardinality 'name' label from all Histogram and Counter metrics across controller, manager, and e2b layers. Only namespace dimension is retained for these metric types to prevent time series explosion in production environments. Affected metrics: - Controller: sandbox_creation/pause/resume/deletion_duration_seconds, sandbox_creation/pause/resume_total - SandboxSet: sandboxset_sandboxes_created/claimed_total - SandboxClaim: sandboxset_claims_total, sandboxclaim_expired_total - Manager: all duration histograms, response/total counters, route sync metrics - E2B: sandbox_snapshot_duration_seconds, sandbox_snapshot_total Preserved with name label: - sandbox_status_abnormal (Gauge) - needs per-sandbox granularity for stuck state detection Also fix test isolation by using unique namespaces per test case to prevent cross-test interference in namespace-level histograms.

furykerry · 2026-05-08T01:33:14Z

 		// Requirement: Track failure in API layer
-		sandboxClaimCreationResponses.WithLabelValues("failure").Inc()
-		sandboxClaimTotal.WithLabelValues("failure", "unknown").Inc()
+		sandboxClaimCreationResponses.WithLabelValues("", "failure").Inc()


we can get namespace from the template or Checkpoint, consider refactor HasTemplate/HasCheckpoint to GetTemplate/GetCheckpoint and return the template/checkpoint object. Consider modify CreateSandbox to save the namespace in the context

furykerry · 2026-05-08T01:48:49Z

 			ConstLabels: prometheus.Labels{"source": "k8s"},
 			Buckets:     prometheus.ExponentialBuckets(0.02, 2, 12), // 20ms -> ~41s
 		},
+		[]string{"namespace", "name"},


plz also remove name label

furykerry · 2026-05-08T01:55:42Z

+}
+
+// DeleteSandboxClaimCounterMetrics is a no-op since counter metrics are at namespace level.
+func DeleteSandboxClaimCounterMetrics(namespace, name string) {


remove no-ops func?

furykerry · 2026-05-08T02:09:55Z

 		}
 	}

+	sandboxStatusAbnormal.WithLabelValues(namespace, name, "pause_incomplete").Set(pauseAbnormal)


can we delete sandboxStatusAbnormal metrics if sandbox becomes healthy again. i think we should only keep abnormal metrics

furykerry · 2026-05-08T02:35:50Z

+			Help:        "Cumulative number of Sandboxes claimed from the SandboxSet",
+			ConstLabels: prometheus.Labels{"source": "k8s"},
+		},
+		[]string{"namespace"},


for sandboxSetSandboxesCreatedTotal and sandboxSetSandboxesClaimedTotal, we can add sandboxset name as label, the number of sandboxset is much less than sandbox and sandboxclaims

furykerry · 2026-05-08T02:37:21Z

+
+var (
+	// sandboxSetClaimsTotal tracks the cumulative number of times a SandboxSet has been claimed.
+	sandboxSetClaimsTotal = prometheus.NewCounterVec(


is it possible to merge this file to pkg/sandboxlcaim/metrics.go?

…point Refactor Infrastructure interface to return full CR objects instead of booleans, enabling proper namespace extraction for Prometheus metric labels. Changes: - HasTemplate(ctx, name) bool -> GetTemplate(ctx, name) (*v1alpha1.SandboxSet, error) - HasCheckpoint(ctx, name) bool -> GetCheckpoint(ctx, name) (*v1alpha1.Checkpoint, error) - ClaimSandbox: use template.GetNamespace() for metric namespace label - CloneSandbox: attempt GetCheckpoint to extract namespace on failure - Update E2B CreateSandbox and all related tests/mocks accordingly

kruise-bot · 2026-05-08T02:51:30Z

@KeyOfSpectator: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

refine sandbox prometheus metric

d4f107f

kruise-bot requested review from AiRanthem, furykerry and zmberg May 6, 2026 08:31

kruise-bot added the size/XXL label May 6, 2026

refine ut

d2a94e6

furykerry reviewed May 6, 2026

View reviewed changes

Comment thread pkg/controller/sandbox/metrics.go Outdated

KeyOfSpectator added 2 commits May 7, 2026 13:37

furykerry reviewed May 8, 2026

View reviewed changes

kruise-bot added the needs-rebase label May 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(metrics): add operation counters, deletion duration, resume type label, and abnormal state detection#336

feat(metrics): add operation counters, deletion duration, resume type label, and abnormal state detection#336
KeyOfSpectator wants to merge 5 commits intoopenkruise:masterfrom
KeyOfSpectator:features/refine-prometheus-metric-20260506

KeyOfSpectator commented May 6, 2026 •

edited

Loading

Uh oh!

kruise-bot commented May 6, 2026

Uh oh!

codecov Bot commented May 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

furykerry May 8, 2026 •

edited

Loading

Uh oh!

furykerry May 8, 2026

Uh oh!

furykerry May 8, 2026

Uh oh!

furykerry May 8, 2026

Uh oh!

furykerry May 8, 2026

Uh oh!

furykerry May 8, 2026

Uh oh!

kruise-bot commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

KeyOfSpectator commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Changes

Label Cardinality Strategy

Controller Layer — New Metrics

Controller Layer — Existing Metrics Enhanced

SandboxSet Layer

SandboxClaim Layer

Manager Layer

E2B Layer

Key Design Decisions

How to verify

Uh oh!

kruise-bot commented May 6, 2026

Uh oh!

codecov Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

furykerry May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

furykerry May 8, 2026

Choose a reason for hiding this comment

Uh oh!

furykerry May 8, 2026

Choose a reason for hiding this comment

Uh oh!

furykerry May 8, 2026

Choose a reason for hiding this comment

Uh oh!

furykerry May 8, 2026

Choose a reason for hiding this comment

Uh oh!

furykerry May 8, 2026

Choose a reason for hiding this comment

Uh oh!

kruise-bot commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KeyOfSpectator commented May 6, 2026 •

edited

Loading

codecov Bot commented May 6, 2026 •

edited

Loading

furykerry May 8, 2026 •

edited

Loading