[Invoker] Wire up ReservedResource to replace the old Permit by AhmedSoliman · Pull Request #4649 · restatedev/restate

AhmedSoliman · 2026-04-24T16:21:52Z

With this change, user-limits acquisition correctly tracks the lifecycle of the invocation state machine on the invoker.

Stack created with Sapling. Best reviewed with ReviewStack.

I'm not loving this, but it works well enough to unblock.

NOTE: sys_user_limits is a *stub* table, the shape and contents of this table will almost definitely change in the very near future. Consider this as a placeholder until scope+limit-keys are wired up completely.

Expose vqueue entries as a single DataFusion table with stage-aware scanning. When the query filters `stage`, only matching stage key kinds are scanned; without a stage filter, all inbox stages are scanned and merged. Also project the latest entry metadata for observability (status plus EntryStatistics timestamps and counters), and add targeted tests for stage predicate extraction and sys_vqueues stage filtering behavior.

Baby steps towards unification and changing the ownership model of the invoker

With this change, user-limits acquisition correctly tracks the lifecycle of the invocation state machine on the invoker.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6bd1d192b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-24T16:36:55Z

+        if self
+            .notified_head
+            .is_some_and(|notified| notified == vqueue)
+        {
+            return Poll::Pending;


Advance throttling head when notified waiter doesn't acquire

poll_head keeps the throttling head in-place and then suppresses further wakeups while notified_head == vqueue, but the acquire path checks other resources (e.g. lock/user limits) before throttling (resource_manager.rs), so a woken queue can get blocked elsewhere and never call poll_acquire to consume/remove its reservation. In that state this branch returns Pending forever for the same head, queues behind it stop making progress, and reserved throttling capacity is stranded until the head happens to re-enter throttling, which can severely stall scheduling under lock+throttle contention.

Useful? React with 👍 / 👎.

github-actions · 2026-04-24T16:49:58Z

Test Results

8 files ±0 8 suites ±0 2m 9s ⏱️ +14s
50 tests ±0 50 ✅ ±0 0 💤 ±0 0 ❌ ±0
218 runs ±0 218 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit b6bd1d1. ± Comparison against base commit bd7dc3c.

github-actions · 2026-04-24T16:49:58Z

Test Results

5 files - 3 5 suites - 3 1m 37s ⏱️ -18s
27 tests - 23 27 ✅ - 23 0 💤 ±0 0 ❌ ±0
54 runs - 164 54 ✅ - 164 0 💤 ±0 0 ❌ ±0

Results for commit b6bd1d1. ± Comparison against base commit bd7dc3c.

This pull request removes 50 and adds 27 tests. Note that renamed tests count towards both.

dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[1]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[2]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[3]
dev.restate.sdktesting.tests.Combinators ‑ awakeableOrTimeoutUsingAwaitAny(Client)
…

dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithFailure(Client)
dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithSuccess(Client)
dev.restate.sdktesting.tests.AwakeableLeaderTransferTest ‑ awakeableCompletionsAreNotLostDuringLeaderTransfer(Client, ContainerHandle)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeSend(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeService(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeVirtualObject(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentSendThenAttachWIthIdempotencyKey(Client)
dev.restate.sdktesting.tests.IngressTest ‑ privateService(URI, Client)
dev.restate.sdktesting.tests.InvokerMemoryTest ‑ allInvocationsCompleteUnderMemoryPressure(Client, URI, int)
dev.restate.sdktesting.tests.InvokerMemoryTest ‑ allStatefulInvocationsCompleteUnderMemoryPressure(Client, URI, int)
…

github-actions · 2026-04-24T16:49:58Z

Test Results

7 files - 1 7 suites - 1 2m 38s ⏱️ +43s
47 tests - 3 47 ✅ - 3 0 💤 ±0 0 ❌ ±0
200 runs - 18 200 ✅ - 18 0 💤 ±0 0 ❌ ±0

Results for commit b6bd1d1. ± Comparison against base commit bd7dc3c.

This pull request removes 7 and adds 4 tests. Note that renamed tests count towards both.

dev.restate.sdktesting.tests.Combinators ‑ awakeableOrTimeoutUsingAwaitAny(Client)
dev.restate.sdktesting.tests.Combinators ‑ firstSuccessfulCompletedAwakeable(Client)
dev.restate.sdktesting.tests.Custom ‑ run(CustomTestConfig, URI, URI)[1]
dev.restate.sdktesting.tests.UserErrors ‑ failSeveralTimesWithMetadata(URI)
dev.restate.sdktesting.tests.UserErrors ‑ internalCallFailurePropagationWithMetadata(URI)
dev.restate.sdktesting.tests.UserErrors ‑ invokeTerminallyFailingCallWithMetadata(URI)
dev.restate.sdktesting.tests.UserErrors ‑ sideEffectWithTerminalErrorWithMetadata(URI)

dev.restate.sdktesting.tests.KafkaIngress ‑ handleEventInCounterService(URI, int, Client)
dev.restate.sdktesting.tests.KafkaIngress ‑ handleEventInEventHandler(URI, int, Client)
dev.restate.sdktesting.tests.UpgradeWithInFlightInvocation ‑ inFlightInvocation(Client, URI)
dev.restate.sdktesting.tests.UpgradeWithNewInvocation ‑ executesNewInvocationWithLatestServiceRevisions(Client, URI)

♻️ This comment has been updated with latest results.

AhmedSoliman added 8 commits April 24, 2026 13:13

[VQueues] Concrete implementation of invoker throttling

6da3278

I'm not loving this, but it works well enough to unblock.

[DataFusion] Unify partition leader status and leader query protocol

e864d2d

[VQueues] Adds sys_scheduler and sys_user_limits tables

1aeeafb

NOTE: sys_user_limits is a *stub* table, the shape and contents of this table will almost definitely change in the very near future. Consider this as a placeholder until scope+limit-keys are wired up completely.

[Invoker] Make service partition-owned and preserve metric labels

42f7147

Merging invoker-api and worker-api into a single crate

8c3da6d

Baby steps towards unification and changing the ownership model of the invoker

Move resource permits into worker-api

f8bf69d

[Invoker] Wire up ReservedResource to replace the old Permit

b6bd1d1

With this change, user-limits acquisition correctly tracks the lifecycle of the invocation state machine on the invoker.

AhmedSoliman requested a review from tillrohrmann April 24, 2026 16:22

chatgpt-codex-connector Bot reviewed Apr 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Invoker] Wire up ReservedResource to replace the old Permit#4649

[Invoker] Wire up ReservedResource to replace the old Permit#4649
AhmedSoliman wants to merge 8 commits intomainfrom
pr4649

AhmedSoliman commented Apr 24, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 24, 2026

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

github-actions Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

AhmedSoliman commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 24, 2026

Test Results

Uh oh!

github-actions Bot commented Apr 24, 2026

Test Results

Uh oh!

github-actions Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

AhmedSoliman commented Apr 24, 2026 •

edited

Loading

github-actions Bot commented Apr 24, 2026 •

edited

Loading