Skip to content

Async runtime for io_uring#472

Draft
proteetpaul wants to merge 26 commits intoXiangpengHao:mainfrom
proteetpaul:io-uring-runtime
Draft

Async runtime for io_uring#472
proteetpaul wants to merge 26 commits intoXiangpengHao:mainfrom
proteetpaul:io-uring-runtime

Conversation

@proteetpaul
Copy link
Copy Markdown
Contributor

This commit introduces a multi-threaded runtime that integrates io_uring. Each worker thread has the following components:

  • A local async executor
  • An io_uring instance

Tasks are submitted to the worker thread using a crossbeam channel, and results are propagated back to the caller using a oneshot channel. When a task requires IO, it will yield to the runtime. The runtime is responsible for submitting IO's to the ring and polling IO completions. Upon completion, the runtime will unpark the corresponding task.

TODO():

  • Batch IO submissions to amortize the syscall overhead
  • Better admission control

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 14, 2026

Codecov Report

❌ Patch coverage is 89.74359% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.89%. Comparing base (f3a8da2) to head (533869e).

Files with missing lines Patch % Lines
src/parquet/src/io/io_uring/tasks.rs 67.85% 5 Missing and 4 partials ⚠️
src/parquet/src/io/io_uring/runtime.rs 95.70% 3 Missing and 4 partials ⚠️
src/common/src/io_mode.rs 0.00% 2 Missing ⚠️
src/parquet/src/io/io_backend.rs 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #472      +/-   ##
==========================================
+ Coverage   83.83%   83.89%   +0.05%     
==========================================
  Files          97       98       +1     
  Lines       21526    21721     +195     
  Branches    21526    21721     +195     
==========================================
+ Hits        18047    18222     +175     
- Misses       3054     3066      +12     
- Partials      425      433       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

proteetpaul and others added 18 commits February 24, 2026 02:31
This commit implements a buffer pool for fixed buffers in order to remove memory allocation
and pinning overheads during io submission. The buffer pool is integrated into the uring
threadpool mechanism. Also implements minor optimizations in the uring threadpool, such as syscall batching.
- Syscall batching
- Handle multiple sqes per IO task
- Spawn a batch of tasks on the runtime
Partition batches more evenly, add support for other io modes, minor code cleanup
- Add error handling in pool.rs
- Run cargo fmt
- Minor fixes
@codacy-production
Copy link
Copy Markdown

codacy-production bot commented Apr 1, 2026

Up to standards ✅

🟢 Issues 49 high · 6 medium

Results:
55 new issues

Category Results
Security 49 high
Complexity 6 medium

View in Codacy

🟢 Metrics 165 complexity · 103 duplication

Metric Results
Complexity 165
Duplication 103

View in Codacy

TIP This summary will be updated as you push new changes. Give us feedback

- Steal tasks only from the global queue
- Basic admission control: Only a fixed number of tasks are admitted per worker
- Table formatting for results
- Use a tokio runtime if io mode is std-blocking
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant