[VQueues] Move scheduler status types to worker-api#4638
Merged
AhmedSoliman merged 2 commits intomainfrom Apr 24, 2026
Merged
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31e4a357a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Test Results 8 files ± 0 8 suites ±0 5m 36s ⏱️ + 3m 38s Results for commit e312a76. ± Comparison against base commit 553ca52. This pull request removes 1 and adds 4 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
AhmedSoliman
force-pushed
the
pr4638
branch
2 times, most recently
from
3915988 to
dec339d
Compare
This was referenced Apr 23, 2026
tillrohrmann
approved these changes
Apr 24, 2026
Contributor
tillrohrmann
left a comment
tillrohrmann
left a comment
There was a problem hiding this comment.
LGTM. +1 for merging :-)
Comment on lines
-93
to
-94
| /// The vqueue is waiting to acquire a lock of a VO. | ||
| BlockedOnLock, |
Contributor
There was a problem hiding this comment.
Is this SchedulingStatus redundant given that we have ResourceKind::Lock?
Contributor
Author
There was a problem hiding this comment.
Removed in subsequent PR
Comment on lines
-95
to
-98
| /// The vqueue is waiting for concurrency tokens. Concurrency tokens are released | ||
| /// when currently running items are completed or (in some cases) when running items | ||
| /// are parked. | ||
| WaitingConcurrencyTokens, |
This was referenced Apr 24, 2026
…time accounting ## sys_vqueue_meta - New partitioned datafusion table surfacing per-vqueue identity, lifecycle flags/timestamps, EMA durations, and per-stage counts from `VQueueMeta`. - New `ScanVQueueMetaTable` trait in `storage-api`, implemented on `PartitionStore` at `Priority::Low`; decode uses `VQueueMetaRef` via bilrost `decode_borrowed` for zero-copy string columns. ## Wait-time accounting - `Stats` rewritten around a single `Option<(WaitBucket, Instant)>` + `EnumMap` accumulator. *At most one open wait segment* is now a structural invariant, fixing over-attribution on resource transitions. - One `set_wait()` entry point; `finalize()` / `snapshot()` share one builder; `reset()` flushes open segments so preempted items still contribute to node-wide counters. - `ResourceKind -> WaitBucket` mapping is exhaustive — new variants are a compile error until a bucket is assigned. - Node-level invoker throttling is one bucket whether observed as `BlockedOn` or as a token-bucket delay. ## EMAs - Two new `VQueueStatistics` fields sampled on **every** `Inbox -> Running` transition (retries included): `avg_blocked_on_concurrency_rules_ms`, `avg_blocked_on_invoker_throttling_ms`. - Carried through `MoveMetrics`; surfaced as two `Duration` columns on `sys_vqueue_meta`. - Aspirational per-entry wait-time todo in `EntryStatistics` removed — vqueue-level EMA preferred over per-entry signal that decays in relevance. ## Naming - `WaitStats` fields uniformly use `blocked_on_*_ms`; metric constants/strings follow `VQUEUE_<X>_WAIT_MS` / `restate.vqueue.scheduler.<x>_wait_ms.total`; `WaitBucket` and `ResourceKind` variants use matching names. - Pre-existing metric-string / describe-text inconsistencies fixed. - Bilrost tag numbers preserved on renamed fields — wire-compatible.
This is to allow datafusion storage engine to access these types without direct dependency on restate-vqueues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is to allow datafusion storage engine to access these types without direct dependency on restate-vqueues.
Stack created with Sapling. Best reviewed with ReviewStack.