Unsigned integer overflow in `toom_eval_pm2.c`

[Niels Möller]

Andrew Teylu <andrewvaughanj at gmail.com> writes:

>> I am not sure the arithmetic on unsigned types is what clang is unhappy
>> about, though.  Perhaps it dislikes the xor with "neg", which is a
>> signed variable.

I can't say precisely what implicit conversions happen according to the
spec: Unsigned to signed is always well defined, but the other direction
only if the converted value fits. It may also depend on whether or not
mp_limb_t is larger than int.

Does it make any difference if you change the "1" constants to "1u" ?

I see no good reason to involve any signed values here, though. Maybe
the variable neg, and the return value, should be changed to mp_limb_t,
or at least unsigned int?

> I guess maybe a different question is: what is that code supposed to
> do? Is the intent to `xor` with `0xFFFFFFFF` if the `k` is even and
> `xor` with `0` if the `k` is odd (i.e., it either flips all the bits
> in the even case or leaves them in the odd case)?

I think intention is to conditionally flip all the bits. And in
addition, neg should always be either all ones or all zeros.

Regards,
/Niels

-- 
Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677.
Internet email is subject to wholesale government surveillance.