Fix handling of otherName in ConfirmNameConstraints

[embhorn]

Description

• ConfirmNameConstraints ignored the otherName GeneralName form (and the extNameConstraintCrit flag was never propagated to Signer)
• Adjust default behavior of using intermediate non-CA certs in wolfSSL_X509_verify_cert with compatibility layer enabled. Added WOLFSSL_X509_STORE_ALLOW_NON_CA_INTERMEDIATE for backward compatibility.

Fixes

• zd21707
• zd21722

Testing

Added test_NameConstraints_OtherName

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR fixes RFC 5280 nameConstraints enforcement for otherName SAN entries and ensures critical nameConstraints behavior is correctly propagated and enforced during verification.

Changes:

• Add byte-exact otherName matching for permitted/excluded nameConstraints subtrees.
• Track and fail-closed on critical nameConstraints that contain unsupported GeneralName forms.
• Propagate extNameConstraintCrit (+ new unsupported-form flag) into Signer, and add a regression test.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
wolfssl/wolfcrypt/asn.h	Adds flags to track critical nameConstraints and presence of unsupported constraint forms in DecodedCert/Signer.
wolfcrypt/src/asn.c	Implements otherName constraint matching, records unsupported forms during decode, propagates flags, and enforces fail-closed behavior for critical constraints.
tests/api.c	Adds a new test that builds a chain using otherName SAN + nameConstraints to verify enforcement.

Comments suppressed due to low confidence (3)

wolfcrypt/src/asn.c:1

• New fail-closed behavior is introduced for critical nameConstraints containing unsupported GeneralName forms, but the added test suite only covers otherName. Please add a regression test that constructs a critical nameConstraints with an unsupported form (e.g., registeredID, x400Address, or ediPartyName) and asserts verification rejects; also add a non-critical variant asserting verification continues (since the fail-closed requirement is for critical constraints).
  wolfcrypt/src/asn.c:1
• The otherName byte-exact matching logic (including the WOLFSSL_FPKI oidSum guard) is duplicated in both PermittedListOk and IsInExcludedList. Consider extracting a small helper (e.g., MatchOtherNameConstraint(name, current)) so future updates don’t accidentally diverge between permitted vs excluded handling.
  wolfcrypt/src/asn.c:1
• The otherName byte-exact matching logic (including the WOLFSSL_FPKI oidSum guard) is duplicated in both PermittedListOk and IsInExcludedList. Consider extracting a small helper (e.g., MatchOtherNameConstraint(name, current)) so future updates don’t accidentally diverge between permitted vs excluded handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10339

Scan targets checked: wolfcrypt-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4

• FLASH: .text +192 B (+0.1%, 197,051 B / 262,144 B, total: 75% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +192 B (+0.3%, 64,499 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +192 B (+0.3%, 60,085 B / 262,144 B, total: 23% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +192 B (+0.2%, 120,506 B / 262,144 B, total: 46% used)

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10339

Scan targets checked: wolfcrypt-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅