Unsigned integer overflow in `toom_eval_pm2.c`

[Andrew Teylu]

On Sun, Sep 3, 2023 at 9:16 PM Niels Möller <nisse at lysator.liu.se> wrote:
> Does it make any difference if you change the "1" constants to "1u" ?
>

I'll try this in the morning and see if the runtime error goes away or not.

> I see no good reason to involve any signed values here, though. Maybe
> the variable neg, and the return value, should be changed to mp_limb_t,
> or at least unsigned int?
>

I don't know much about the GMP code, but looking at this code and its
surroundings, it seems that `neg` is actually `sign` (well, how it is
used by its callees).

I guess maybe rather than "negation" or "negated" (i.e., the
"opposite" of some value), it actually means "is_negative" (==
"sign"). In that case, I guess it doesn't matter if the return value
is signed or not: it is effectively just a bool with two values (0x0
and 0xFFFFFFFF).