We are currently migrating Bugzilla to GitHub issues.
Any changes made to the bug tracker now will be lost, so please do not post new bugs or make changes to them.
When we're done, all bug URLs will redirect to their equivalent location on the new bug tracker.

Bug 4148 - SDL_RLEsurface may encode RGB888 if it has no color key
Summary: SDL_RLEsurface may encode RGB888 if it has no color key
Status: NEW
Alias: None
Product: SDL
Classification: Unclassified
Component: video (show other bugs)
Version: HG 2.1
Hardware: All All
: P2 normal
Assignee: Sam Lantinga
QA Contact: Sam Lantinga
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-24 13:10 UTC by Simon Hug
Modified: 2018-04-24 13:12 UTC (History)
0 users

See Also:

Attachments
Patch that fixes check if surfaces have color keys or the BLEND mode with an alpha channel. (484 bytes, patch)
2018-04-24 13:10 UTC, Simon Hug 		Details | Diff
Test case that blits an RGB888 surfaces with the BLEND blend mode and the RLE hint. (3.54 KB, text/x-csrc)
2018-04-24 13:12 UTC, Simon Hug 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Hug 2018-04-24 13:10:49 UTC 
Created attachment 3228 [details]
Patch that fixes check if surfaces have color keys or the BLEND mode with an alpha channel.

The function SDL_RLESurface has some checks that test whether the surface is eligible for run-length encoding. There seems to be an error on line 1424 [1] where the function returns when there's no actual transparent pixels to encode. With the current check, a RGB888 texture with the BLEND blend mode passes this test. It then goes on the encode the data with the assumption that it has a valid color key, making the pixels with the value of the colorkey member transparent.

I'm not that familiar with the RLE code, but it seems to me that it should also check if the surface has an alpha channel if the BLEND mode is set. This is what the attached patch does.

[1] https://hg.libsdl.org/SDL/file/bc2aba33ae1f/src/video/SDL_RLEaccel.c#l1424
Comment 1 Simon Hug 2018-04-24 13:12:33 UTC 
Created attachment 3229 [details]
Test case that blits an RGB888 surfaces with the BLEND blend mode and the RLE hint.

Small test case which shows the issue.