openssl compat: fix ASN1_STRING_{length,get0_data} for ASN1_INTEGER

[julek-wolfssl]

In OpenSSL, ASN1_INTEGER is typedef'd to ASN1_STRING (same struct), so
calling ASN1_STRING_length() / ASN1_STRING_get0_data() on an
ASN1_INTEGER* is valid and well-defined. wolfSSL has them as distinct,
incompatible structs. This fixes the openvpn master failures introduced in
OpenVPN/openvpn#1003

[Copilot]

Pull request overview

Improves OpenSSL-compatibility for ASN1_INTEGER by allowing ASN1_STRING_length() / ASN1_STRING_get0_data() call sites (common in OpenSSL-based consumers like OpenVPN) to work correctly with wolfSSL’s distinct WOLFSSL_ASN1_INTEGER type.

Changes:

• Add wolfSSL_ASN1_INTEGER_get_length() and wolfSSL_ASN1_INTEGER_get0_data() APIs.
• Implement the new INTEGER helpers in src/ssl_asn1.c (attempting to strip DER tag/length when present).
• Override ASN1_STRING_length() / ASN1_STRING_get0_data() in the OpenSSL compatibility header using C11 _Generic dispatch for INTEGER.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
wolfssl/ssl.h	Declares new INTEGER helper APIs to support compat dispatch.
wolfssl/openssl/ssl.h	Adds C11 _Generic macros to route ASN1_STRING_length/get0_data calls to INTEGER-aware helpers.
src/ssl_asn1.c	Implements INTEGER length/data helpers that attempt to return raw value octets (DER header stripped when detected).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lealem47]

Jenkins retest this please.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10089

Scan targets checked: src, src-bugs, src-compliance
Findings: 1

High (1)

Incorrect GetLength return value check makes DER-stripping logic dead code

File: src/ssl_asn1.c:1017-1021 and 1041-1045
Function: wolfSSL_ASN1_INTEGER_get_length / wolfSSL_ASN1_INTEGER_get0_data
Category: Logic errors

Both wolfSSL_ASN1_INTEGER_get_length and wolfSSL_ASN1_INTEGER_get0_data check GetLength(ai->data, &idx, &len, (word32)ai->length) > 0 to determine if the DER tag/length header was successfully parsed. However, wolfSSL's GetLength function returns 0 on success (storing the decoded length in the *len output parameter) and returns negative error codes on failure. It does not return positive values. Because the check is > 0 instead of >= 0, the condition is never true on a successful parse, making the entire DER-header-stripping branch unreachable dead code. Both functions will always fall through to the fallback path that returns ai->length or ai->data unchanged, completely defeating the stated purpose of the PR (stripping the DER header from ASN1_INTEGER data for OpenSSL compatibility).

if (GetLength(ai->data, &idx, &len, (word32)ai->length) > 0 &&
        idx + (word32)len == (word32)ai->length) {
    return len;
}

Recommendation: Change > 0 to >= 0 in both functions:

if (GetLength(ai->data, &idx, &len, (word32)ai->length) >= 0 &&
        idx + (word32)len == (word32)ai->length) {

---

This review was generated automatically by Fenrir. Findings are non-blocking.

[padelsbach]

@julek-wolfssl, I'm following this PR because I believe it solves the issues in CI. Any thoughts on the Fenrir comments?

[douzzer]

Thanks @julek-wolfssl !

[julek-wolfssl]

@padelsbach comment reasoning was flawed but conclusion is correct. Addressed in #10109.