target/i386: fix fxam handling of invalid encodings

The fxam implementation does not check for invalid encodings, instead
treating them like NaN or normal numbers depending on the exponent.
Fix it to check that the high bit of the significand is set before
treating an encoding as NaN or normal, thus resulting in correct
handling (all of C0, C2 and C3 cleared) for invalid encodings.

Backports commit 34b9cc076ff423023a779a04a9f7cd7c17372cbf from qemu
This commit is contained in:
Joseph Myers 2020-06-15 13:07:52 -04:00 committed by Lioncash
parent 5a01ea31eb
commit c796ee5e13

View file

@ -1065,7 +1065,7 @@ void helper_fxam_ST0(CPUX86State *env)
if (expdif == MAXEXPD) {
if (MANTD(temp) == 0x8000000000000000ULL) {
env->fpus |= 0x500; /* Infinity */
} else {
} else if (MANTD(temp) & 0x8000000000000000ULL) {
env->fpus |= 0x100; /* NaN */
}
} else if (expdif == 0) {
@ -1074,7 +1074,7 @@ void helper_fxam_ST0(CPUX86State *env)
} else {
env->fpus |= 0x4400; /* Denormal */
}
} else {
} else if (MANTD(temp) & 0x8000000000000000ULL) {
env->fpus |= 0x400;
}
}