As an addendum to this, and as a reminder: SDL_WINDOW_FULLSCREEN_DESKTOP is not quite the same as an SDL_WINDOW_RESIZABLE window that's been pushed to a Space; everything is the same except for the inability to use the titlebar when in FULLSCREEN_DESKTOP mode.

So we have various test scenarios:

- Start with a FULLSCREEN_DESKTOP window
- Cmd+Ctrl+F, freeze

- Start with a window that's resizable
- Cmd+Ctrl+F to fullscreen
- Cmd+Ctrl+F back to windowed mode, all is well

- Start with a window that's resizable
- Toggle fullscreen to FULLSCREEN_DESKTOP
- Cmd+Ctrl+F back to windowed mode, freeze

- Start with a window that's resizable
- Cmd+Ctrl+F to fullscreen
- Toggle fullscreen to FULLSCREEN_DESKTOP, no changes
- Cmd+Ctrl+F, no deadlock but window border is missing
- Toggle back to windowed mode, border still missing
- Go back to fullscreen however you want, everything's normal again

If there's a way we can disable Cmd+Ctrl+F on SDL's end we could implement this on the application side to just use our own fullscreen toggle, but on the other hand it'd be nice to have the fullscreen-with-titlebar behavior in all fullscreen scenarios with the application checking for user fullscreen toggle events.

I tried reproducing this with testsprite2 on Mac OS X 10.12.6, and Cmd-Ctrl-F did nothing and I couldn't interact with the View menu. Alt-Enter does toggle fullscreen desktop mode, so I used that and it appeared to work correctly when testsprite2 was not resizable.

Looking at the code, it looks like it attempts to handle this:
In windowWillEnterFullScreen() it sets the NSWindowStyleMaskResizable flag on the window, and then clears it when entering fullscreen is complete.

I did more complete testing with testsprite2, where Ctrl-Enter toggles fullscreen, Alt-Enter toggles fullscreen-desktop and Shift-Enter toggles between fullscreen and fullscreen-desktop.

Here are the results with your test scenarios:
- Start with a FULLSCREEN_DESKTOP window
- Shift-Enter to FULLSCREEN
= transition out of fullscreen desktop, switch to fullscreen, all is well

- Start with a window that's resizable
- Ctrl-Enter to fullscreen
- Ctrl-Enter back to windowed mode 
= switched fullscreen and back, all is well

- Start with a window that's resizable
- Alt-Enter to FULLSCREEN_DESKTOP
- Alt-Enter back to windowed mode
= switched fullscreen and back, all is well

- Start with a window that's resizable
- Ctrl-Enter to fullscreen
- Shift-Enter to FULLSCREEN_DESKTOP
- Shift-Enter to FULLSCREEN
- Ctrl-Enter back to windowed mode
- Alt-Enter back to FULLSCREEN_DESKTOP
= all transitions worked as expected

Are you still able to reproduce this with the latest SDL snapshot?
http://www.libsdl.org/tmp/SDL-2.0.zip

If so, can you attach a minimal test that shows the problem?

Thanks!

Just tried with rev 381449ca1cd2 and it looks like there's a different problem altogether, I'll try and sidestep it locally but pasting here in case nobody else has hit this yet:


* thread #1, name = 'MainThrd', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x00000001008731c0 libSDL2-2.0.0.dylib`UpdateKeymap(data=0x0000000000000000, send_event=SDL_FALSE) at SDL_cocoakeyboard.m:515 [opt]
   512 	
   513 	    /* See if the keymap needs to be updated */
   514 	    key_layout = TISCopyCurrentKeyboardLayoutInputSource();
-> 515 	    if (key_layout == data->key_layout) {
   516 	        return;
   517 	    }
   518 	    data->key_layout = key_layout;
(lldb) bt
* thread #1, name = 'MainThrd', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x00000001008731c0 libSDL2-2.0.0.dylib`UpdateKeymap(data=0x0000000000000000, send_event=SDL_FALSE) at SDL_cocoakeyboard.m:515 [opt]
    frame #1: 0x0000000100872fd0 libSDL2-2.0.0.dylib`Cocoa_InitKeyboard(_this=<unavailable>) at SDL_cocoakeyboard.m:576 [opt]
    frame #2: 0x0000000100876ea2 libSDL2-2.0.0.dylib`Cocoa_VideoInit(_this=0x0000000101a62600) at SDL_cocoavideo.m:159 [opt]
    frame #3: 0x000000010086a192 libSDL2-2.0.0.dylib`SDL_VideoInit_REAL(driver_name=<unavailable>) at SDL_video.c:532 [opt]
    frame #4: 0x00000001007df81a libSDL2-2.0.0.dylib`SDL_InitSubSystem_REAL(flags=<unavailable>) at SDL.c:170 [opt]
    frame #5: 0x00000001001f70aa WonderBoy`Pasta::SDLApplication::SDLApplication(bool) + 90
    frame #6: 0x000000010011f79a WonderBoy`WbGameApp::WbGameApp() + 42
    frame #7: 0x000000010011f641 WonderBoy`main + 353
    frame #8: 0x00007fffaf66d235 libdyld.dylib`start + 1

Whuh, rebuilt some files while debugging and the problem went away. So that's nice.

The good news is that all of the cases as of the current revision are fixed, but it looks like there's one more that's been added:

- Start with a FULLSCREEN_DESKTOP window
- Cmd+Ctrl+F, window works!
- Cmd+Ctrl+F, window disappears for... quite a long time. In fact, it's exactly 37 seconds every time from what I can tell?!
- Fullscreen eventually shows up and works!

So somewhere there's a sleep or a retry timeout that we have to remove or sidestep for that last half of the scenario.

Created attachment 2902 [details]
Test Case

Here's a minimal test case. The window doesn't disappear like my real-world case does (Wonder Boy: The Dragon's Trap) but it does still have that precise timing when going back to fullscreen.

This appears to be a bug in the menu system. If I modify your example to toggle fullscreen state based on keypress it works fine. The first time I ran it I was able to toggle out of fullscreen-desktop state with Cmd+Ctrl+F and then toggling it again with Cmd+Ctrl+F hung the way you described. Subsequent toggles of Cmd+Ctrl+F didn't do anything, and never worked again here, no matter how many times I restarted the test program.

I also can't interact with the Apple menu or the application menus at all, so I'm guessing that we're missing pumping some part of the Cocoa event system.

Alex, can you take a look?

I believe I've fixed the reported issue via https://hg.libsdl.org/SDL/rev/fffd7c61e077 .

The menu bar is completely unclickable for me when I compile and run SDL's test program code (testgl2 etc.) but I haven't been able to reproduce it in my own (simple or otherwise) code. testgl2 often doesn't focus the window properly when its created either - it definitely feels like events are being dropped or something, but I haven't been able to figure out what's causing it.

This fixes all the scenarios I could find, thanks Alex!