macOS Catalina appears to not call the 'IOHIDDeviceRegisterRemovalCallback' callback.
https://forums.developer.apple.com/thread/124444

When you unplug a joystick/gamepad, SDL does not receive a notification, and then subsequently crashes when trying to use the IOHIDDeviceRef.

I was able to work around this by registering for a device removal callback on the manager, which still gets called.

In darwin/SDL_sysjoystick.c, I added a device removal callback:

In 'ConfigHIDManager' function:
IOHIDManagerRegisterDeviceRemovalCallback(hidman, JoystickManagerDeviceWasRemovedCallback, NULL);

Then, I added the callback, like this:
static void
JoystickManagerDeviceWasRemovedCallback(void *ctx, IOReturn res, void *sender, IOHIDDeviceRef ioHIDDeviceObject)
{
    recDevice *curdevice = gpDeviceList;
    while ( curdevice )
    {
        if (curdevice->deviceRef == ioHIDDeviceObject)
        {
            IOHIDDeviceRegisterRemovalCallback(ioHIDDeviceObject, NULL, NULL);
            JoystickDeviceWasRemovedCallback(curdevice, res, sender);
            break;
        }
        curdevice = curdevice->pNext;
    }
}

The code works on both Catalina and older systems.