Segfault in mpn_toom42_mul with small unbalanced inputs (e.g., 9x5) when MUL_TOOM22_THRESHOLD is low

[marco.bodrato at tutanota.com]

Ciao,
16 feb 2026, 10:22 da subhrajit.das at iitgn.ac.in:

> Hi GMP Team,
>
> I have found a bug in GMP 6.3.0 while experimenting with a generic C build
> (--disable-assembly) and attempting to use a lower MUL_TOOM22_THRESHOLD
> (value 5). The issue involves `mpn_toom42_mul` crashing with small
> unbalanced inputs.
>

The current version of gmp-impl.h contains a sequence of #define MPN_TOOMxy_MUL_MINSIZE, and in particular:

#define MPN_TOOM22_MUL_MINSIZE    6
#define MPN_TOOM42_MUL_MINSIZE   10

This means that we already know that the code does not work for some "extreme"  choices of the thresholds.
We usually avoid to add checks resulting in unused branches in the library.
Should we add lines like the following in all toom*.c functions?ASSERT( MUL_TOOM22_THRESHOLD >= MPN_TOOM22_MUL_MINSIZE); 


> ## 1. GMP Version
> GMP 6.3.0 (Patched locally to lower MUL_TOOM22_THRESHOLD to 5).
> Note: We were experimenting with a generic C build (--disable-assembly) and
> attempting to patch gmp-impl.h to use a lower MUL_TOOM22_THRESHOLD (value
> 5) to enable specific optimizations for small sizes. This bug is exposed by
> tuning the threshold, but indicates a logic-check handling in `toom42`
> logic for specific geometries.
>
If you will find thresholds values that could be generated by the speed/tuneup program and that trigger this logic, we will be happy to search for a fix. But if the only way to trigger it is to set a threshold value that is outside of the supported range, then we will keep on maintaining the library as simple and efficient as it can be.

Thanks for testing the library,
mb