SDL/src/render
Sam Lantinga 36998b823e Fixed bug 3689 - MMX YUV renderer crash
felix

The functions in src/render/SDL_yuv_mmx.c contain the following inline assembly snippet:

        /* tap dance to workaround the inability to use %%ebx at will... */
        /*  move one thing to the stack... */
        "pushl $0\n"  /* save a slot on the stack. */
        "pushl %%ebx\n"  /* save %%ebx. */
        "movl %0, %%ebx\n"  /* put the thing in ebx. */
        "movl %%ebx,4(%%esp)\n"  /* put the thing in the stack slot. */
        "popl %%ebx\n"  /* get back %%ebx (the PIC register). */

Here's how it ended up in a binary on my old laptop:

   0xb5c17dbd <ColorRGBDitherYV12MMX1X+93>:	push   $0x0
   0xb5c17dbf <ColorRGBDitherYV12MMX1X+95>:	push   %ebx
   0xb5c17dc0 <ColorRGBDitherYV12MMX1X+96>:	mov    0xc(%esp),%ebx
   0xb5c17dc4 <ColorRGBDitherYV12MMX1X+100>:	mov    %ebx,0x4(%esp)
   0xb5c17dc8 <ColorRGBDitherYV12MMX1X+104>:	pop    %ebx

Apparently the compiler, oblivious to the fact that the assembly snippet manipulates the %esp register, decided to refer to the operand via that same register instead of via %ebp (I believe -fomit-frame-pointer enables this). This causes %ebx to be loaded with the wrong value, which later leads to a null pointer dereference.

Recent GCC can use the %ebx register normally: <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47602#c16>. There is even an explicit constraint "b" for allocating it.
2017-07-20 10:48:57 -07:00
..
direct3d Updated copyright for 2017 2017-01-01 18:33:28 -08:00
direct3d11 WinRT: removed buildbot debug code, pending further research 2017-02-24 20:49:14 -05:00
opengl render: GL_DestroyRender() should activate first. 2017-05-18 21:00:11 -04:00
opengles Removed newlines from error messages. 2017-03-26 21:00:19 +02:00
opengles2 Corrected names of header file guards. 2017-06-03 23:00:15 +02:00
psp Updated copyright for 2017 2017-01-01 18:33:28 -08:00
software Updated copyright for 2017 2017-01-01 18:33:28 -08:00
mmx.h Fixed crash if initialization of EGL failed but was tried again later. 2015-06-21 17:33:46 +02:00
SDL_d3dmath.c Updated copyright for 2017 2017-01-01 18:33:28 -08:00
SDL_d3dmath.h Updated copyright for 2017 2017-01-01 18:33:28 -08:00
SDL_render.c Merged Eric Wing's overscan patch. 2017-06-06 14:06:40 -04:00
SDL_sysrender.h Removed a bunch of unnecessary #ifdefs. 2017-02-26 00:40:04 -05:00
SDL_yuv_mmx.c Fixed bug 3689 - MMX YUV renderer crash 2017-07-20 10:48:57 -07:00
SDL_yuv_sw.c Updated copyright for 2017 2017-01-01 18:33:28 -08:00
SDL_yuv_sw_c.h Updated copyright for 2017 2017-01-01 18:33:28 -08:00