We are currently migrating Bugzilla to GitHub issues.
Any changes made to the bug tracker now will be lost, so please do not post new bugs or make changes to them.
When we're done, all bug URLs will redirect to their equivalent location on the new bug tracker.

Bug 4148

Summary: SDL_RLEsurface may encode RGB888 if it has no color key
Product: SDL Reporter: Simon Hug <chli.hug>
Component: videoAssignee: Sam Lantinga <slouken>
Status: NEW --- QA Contact: Sam Lantinga <slouken>
Severity: normal    
Priority: P2    
Version: HG 2.1   
Hardware: All   
OS: All   
Attachments: Patch that fixes check if surfaces have color keys or the BLEND mode with an alpha channel.
Test case that blits an RGB888 surfaces with the BLEND blend mode and the RLE hint.

Description Simon Hug 2018-04-24 13:10:49 UTC 
Created attachment 3228 [details]
Patch that fixes check if surfaces have color keys or the BLEND mode with an alpha channel.

The function SDL_RLESurface has some checks that test whether the surface is eligible for run-length encoding. There seems to be an error on line 1424 [1] where the function returns when there's no actual transparent pixels to encode. With the current check, a RGB888 texture with the BLEND blend mode passes this test. It then goes on the encode the data with the assumption that it has a valid color key, making the pixels with the value of the colorkey member transparent.

I'm not that familiar with the RLE code, but it seems to me that it should also check if the surface has an alpha channel if the BLEND mode is set. This is what the attached patch does.

[1] https://hg.libsdl.org/SDL/file/bc2aba33ae1f/src/video/SDL_RLEaccel.c#l1424
Comment 1 Simon Hug 2018-04-24 13:12:33 UTC 
Created attachment 3229 [details]
Test case that blits an RGB888 surfaces with the BLEND blend mode and the RLE hint.

Small test case which shows the issue.