We are currently migrating Bugzilla to GitHub issues.
Any changes made to the bug tracker now will be lost, so please do not post new bugs or make changes to them.
When we're done, all bug URLs will redirect to their equivalent location on the new bug tracker.

Bug 3367 - RGBA_FROM_PIXEL macro can't handle SDL_PIXELFORMAT_ARGB2101010
Summary: RGBA_FROM_PIXEL macro can't handle SDL_PIXELFORMAT_ARGB2101010
Status: ASSIGNED
Alias: None
Product: SDL
Classification: Unclassified
Component: video (show other bugs)
Version: HG 2.0
Hardware: All All
: P2 normal
Assignee: Sam Lantinga
QA Contact: Sam Lantinga
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-06-19 17:47 UTC by Simon Hug
Modified: 2017-08-12 01:57 UTC (History)
0 users

See Also:

Attachments
Test case for SDL_PIXELFORMAT_ARGB2101010 blitting. (2.43 KB, text/plain)
2016-06-19 17:47 UTC, Simon Hug 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Hug 2016-06-19 17:47:30 UTC 
Created attachment 2501 [details]
Test case for SDL_PIXELFORMAT_ARGB2101010 blitting.

The RGBA_FROM_PIXEL macro in src/video/blit.h [1] is not designed to work with more than 8 bits per channel and the ARGB2101010 format makes it read outside of the array bounds causing access violations. This can happen during blitting with the BlitNtoNPixelAlpha and SDL_Blit_Slow functions.

When SDL_InitFormat tries to calculate the loss of the channels [2], the Uint8 will wrap around and it will end up at 254 for the 10-bit channels. Clearly way over the 9 entries of the SDL_expand_byte array. (Not that a signed integer would help.) Then the macro tries to access the lookup table with the channel value which could be up to 1023. If the previous indirection didn't cause an access violation this one will.

I guess it's not worth modifying this macro for a format that only a few will use. It will only make the other blitters slower. I don't have good ideas to solve this issue.

Attached is a test case that does three blits. A copy one that work and the two that use the functions mentioned above.

[1] https://hg.libsdl.org/SDL/file/b82c0f22d22a/src/video/SDL_blit.h#l303
[2] https://hg.libsdl.org/SDL/file/b82c0f22d22a/src/video/SDL_pixels.c#l540
Comment 1 Sam Lantinga 2017-08-12 01:57:15 UTC 
I fixed the crash for now:
https://hg.libsdl.org/SDL/rev/7f429e943ed3

That blit combination still isn't supported, but we can add support later if it's needed.