Fix potential mismatch in size between DECL_MP_INT_SIZE_DYN and NEW_MP_INT_SIZE, fix unused variable warning in random.c.

[kareem-wolfssl]

Description

Fixes zd#21858, #10499

Testing

Built in tests

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR addresses potential size mismatches between stack-allocated SP-math mp_int buffers and the amount of memory cleared/initialized, adds explicit key-size bounds checks in RSA/ECC paths to fail fast on oversized moduli, and fixes a build-configuration-specific unused-variable warning in RNG initialization.

Changes:

• Update NEW_MP_INT_SIZE() (SP math, static stack mode) to zero based on the declared digit buffer size to avoid memset exceeding the allocated buffer when DECL_MP_INT_SIZE_DYN(..., bits, max) uses max.
• Add mp_bitsused(...) > {RSA_MAX_SIZE, MAX_ECC_BITS_USE} guards in multiple RSA/ECC code paths and propagate wc_RsaEncryptSize() errors in wc_RsaDirect().
• Fix random.c so HASHDRBG seed variables are not declared when CUSTOM_RAND_GENERATE_BLOCK is enabled, and add an API test covering oversized RSA modulus handling.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
wolfssl/wolfcrypt/sp_int.h	Fix SP-math static-stack clearing to respect the actual declared digit buffer size.
wolfcrypt/src/rsa.c	Add oversized-modulus guards in several RSA paths and propagate wc_RsaEncryptSize() failures.
wolfcrypt/src/random.c	Adjust preprocessor guards to eliminate unused-variable warnings under CUSTOM_RAND_GENERATE_BLOCK.
wolfcrypt/src/ecc.c	Add oversized-modulus/key-size guards across several ECC arithmetic/sign/verify paths.
tests/api/test_rsa.h	Register new RSA oversized-modulus test.
tests/api/test_rsa.c	Add test ensuring oversized modulus returns WC_KEY_SIZE_E (and doesn’t overflow stack buffers).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4

• FLASH: .text +320 B (+0.2%, 197,889 B / 262,144 B, total: 75% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +64 B (+0.1%, 64,955 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +64 B (+0.1%, 60,477 B / 262,144 B, total: 23% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +64 B (+0.1%, 121,117 B / 262,144 B, total: 46% used)