ASN: improve handling of ASN.1 parsing/encoding

ToTraditionalInline_ex2 original ASN code:
  - Now return 0 when no OCTECT_STRING data found.
  - Change callers to accept 0 as a valid returnb value.

SizeASN_Items:
  - Change encoded size to word32 as won't be negative.
- Change callers to supply a pointer to a word32 instead of integer.
Fix casting due to change of parameter type.

ASN_LEN_ENC_LEN: Function to calculate the length of the encoded ASN.1
length.

GetLength_ex:
  - Change minLen to word32
- Change length to word32 and change negative check appropriately for
different type.

GetASNHeader_ex:
  - If not checking lengths in GetLength_ex, check it here.
DecodeObjectId:
  - Ensure no overflow in calculation.

_RsaPrivateKeyDecode (original)
  - Clear RSA integers on failure (will be done in free anyway).

wc_CreatePKCS8Key (original):
  - safe check of overflow.

DecryptContent (templare):
- Parse will fail if OID not recognized, and recognized OIDs are 9/10
bytes long - but check idx is 9/10 anyway so we know we can read 2 end
bytes of data.

wc_RsaPublicKeyDecode_ex (original):
- Fix calculation of seqEndIdx and use it to bound modulus and
exponent.

DecodePolicyOID
  - enusre inSz is not too long.
  - Ensure no overflow in calculation.

SetOidValue (orginal):
  - Safe check of inSz and oidSz.

SetAltNames (original):
  - Improve length checks

FlattenAltNames:
  - Check for overflow.
  - Better length check.

ParseCRL_CertList (original):
  - overflow check

Author: SparkiDev

src/pk.c

@@ -1829,7 +1829,7 @@ int wolfSSL_RSA_LoadDer_ex(WOLFSSL_RSA* rsa, const unsigned char* derBuf,
          * have a PKCS8 header then do not error out. */
         res = ToTraditionalInline_ex((const byte*)derBuf, &idx, (word32)derSz,
             &algId);
-        if (res > 0) {
+        if (res >= 0) {
             /* Store size of PKCS#8 header for encoding. */
             WOLFSSL_MSG("Found PKCS8 header");
             rsa->pkcs8HeaderSz = (word16)idx;
@@ -12452,7 +12452,7 @@ int wolfSSL_EC_KEY_LoadDer_ex(WOLFSSL_EC_KEY* key, const unsigned char* derBuf,
          * have a PKCS8 header then do not error out.
          */
         if ((ret = ToTraditionalInline_ex((const byte*)derBuf, &idx,
-                (word32)derSz, &algId)) > 0) {
+                (word32)derSz, &algId)) >= 0) {
             WOLFSSL_MSG("Found PKCS8 header");
             key->pkcs8HeaderSz = (word16)idx;
             res = 1;

src/ssl.c

@@ -8654,7 +8654,7 @@ static WOLFSSL_EVP_PKEY* _d2i_PublicKey(int type, WOLFSSL_EVP_PKEY** out,
         /* Check if input buffer has PKCS8 header. In the case that it does not
          * have a PKCS8 header then do not error out. */
         if ((ret = ToTraditionalInline_ex((const byte*)(*in), &idx,
-                                          (word32)inSz, &algId)) > 0) {
+                                          (word32)inSz, &algId)) >= 0) {
             WOLFSSL_MSG("Found PKCS8 header");
             pkcs8HeaderSz = (word16)idx;