target/i386: fix fscale handling of invalid exponent encodings

The fscale implementation does not check for invalid encodings in the
exponent operand, thus treating them like INT_MIN (the value returned
for invalid encodings by floatx80_to_int32_round_to_zero). Fix it to
treat them similarly to signaling NaN exponents, thus generating a
quiet NaN result.

Backports commit b40eec96b26028b68c3594fbf34b6d6f029df26a from qemu
This commit is contained in:
Joseph Myers 2020-06-15 13:03:52 -04:00 committed by Lioncash
parent d96c218664
commit bbbf25fdd9

View file

@ -934,7 +934,10 @@ void helper_frndint(CPUX86State *env)
void helper_fscale(CPUX86State *env)
{
if (floatx80_is_any_nan(ST1)) {
if (floatx80_invalid_encoding(ST1)) {
float_raise(float_flag_invalid, &env->fp_status);
ST0 = floatx80_default_nan(&env->fp_status);
} else if (floatx80_is_any_nan(ST1)) {
ST0 = ST1;
if (floatx80_is_signaling_nan(ST0, &env->fp_status)) {
float_raise(float_flag_invalid, &env->fp_status);