Dtls13: ack management improvements

[rizlik]

Improve DTLS 1.3 ACK record handling, enforcing record count limits. It also introduces comprehensive tests for these edge cases.

• Added a strict upper bound (DTLS13_ACK_MAX_RECORDS) on the number of ACK records that can be tracked and encoded

[Copilot]

Pull request overview

Improves DTLS 1.3 ACK handling by adding explicit tracking/counting of seen ACK record numbers, enforcing an upper bound during ACK list building, and adding new API tests to cover overflow and write-dup edge cases.

Changes:

• Add DTLS13_ACK_MAX_RECORDS / DTLS13_RN_SIZE constants and track seenRecordsCount alongside the ACK record list.
• Enforce ACK-list size constraints in Dtls13RtxAddAck() and Dtls13WriteAckMessage(), and reset the counter on flush/transfer paths.
• Add new DTLS 1.3 ACK edge-case tests (overflow behavior and write-dup counter reset).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
wolfssl/internal.h	Adds ACK sizing macros, adds seenRecordsCount to DTLS13 RTX state, and updates DTLS13 internal/test-visible function declarations.
src/dtls13.c	Implements count-based ACK list limiting/encoding and resets counter on flush/transfer/removal paths.
tests/api/test_dtls.h	Registers new DTLS 1.3 ACK tests in the API test list.
tests/api/test_dtls.c	Adds tests for ACK overflow bounds and write-dup counter reset behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rizlik]

retest this please

[rizlik]

retest this please

[rizlik]

retest this please

[programsurf]

Verified this fixes the vulnerability I reported.

• Dtls13GetAckListLength() (word16 truncation) fully removed
• Three independent guards: insert-time cap (128), write-time validation, in-loop bounds check
• Counter stays consistent across insert / flush / remove / write-dup transfer
• Mutex early-return fixes are a nice bonus

Original PoC (4097 records -> 65KB heap overflow) no longer reproduces. LGTM.

— Sunwoo Lee and Seunghyun Yoon, Korea Institute of Energy Technology (KENTECH)

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rizlik]

retest this please