This bug report was migrated from our old Bugzilla tracker.

These attachments are available in the static archive:

• More accurate resampling (SDL_audiotypecvt.patch, text/plain, 2016-12-18 15:21:40 +0000, 8291 bytes)

Reported in version: 2.0.0
Reported for operating system, platform: All, All

Comments on the original bug report:

On 2006-01-20 08:30:09 +0000, Sam Lantinga wrote:

Date: Thu, 19 Jan 2006 17:12:56 -0500
From: "Ryan C. Gordon" icculus@icculus.org
Subject: Re: [SDL] Re: (Time) Length of SDL_mixer chunks?

Second, many hardware drivers only allow power of two buffers for audio DMA,
so the final output rate would change the size of the input buffer, instead
of it being possible to calculate as a function of the input format.

But, if SDL_mixer is as good an example as any: SDL_ConvertAudio() has
uses outside of the audio callback. Perhaps we should remove the
limitation generally and have SDL_OpenAudio() fail if the sample rate
isn't a power of two? Maybe have SDL buffer data that would overflow the
hardware fragment size and give the audio callback a variable-sized
buffer when it has to make up the difference?

It seems that we should probably make this work cleanly behind the
scenes or at least give an error. Right now, people just get strange
results without any explanation as to why.

--ryan.

On 2006-01-20 08:31:22 +0000, Sam Lantinga wrote:

Let's say the audio driver gave you a DMA buffer of 4096 bytes of 16-bit
stereo audio at 48KHz, and you're trying to fill it with 16-bit stereo
audio at 44.1KHz... How many samples should you request in the callback?

On 2006-01-20 08:32:52 +0000, Sam Lantinga wrote:

Date: Fri, 20 Jan 2006 09:49:19 +0100
From: David Olofson david@olofson.net
Subject: Re: [SDL] Re: (Time) Length of SDL_mixer chunks?

44100 / 48000 * 4096 ==> alternating between 3763 atd 3764. (SDL
should report the maximum value, for application side buffer
allocation.)

For interpolation, you'll also need to buffer a few samples
internally, for overlap.

//David Olofson - Programmer, Composer, Open Source Advocate

On 2006-01-20 08:46:42 +0000, Sam Lantinga wrote:

Should we always call back the application with full buffer sizes at the requested format, and then keep around any extra data for the next hardware callback?

e.g.
The app requests 4096 sample buffer size at 44.1KHz, driver provides 1024 sample size at 48KHz. We do the following loop:

driver callback:
fill app sound buffer with 4096 samples
consume 940 samples filling 1024 sample sound buffer at 48KHz, leaving 3156
driver callback:
consume 940 samples filling 1024 sample sound buffer at 48KHz, leaving 2216
driver callback:
consume 940 samples filling 1024 sample sound buffer at 48KHz, leaving 1276
driver callback:
consume 940 samples filling 1024 sample sound buffer at 48KHz, leaving 336
driver callback:
consume 336 samples filling 365 samples at 48KHz
fill app sound buffer with 4096 samples
consume 605 samples filling remaining sound buffer at 48KHz, leaving 3491
...

On 2006-01-20 10:05:54 +0000, David Olofson wrote:

(In reply to comment # 3)

Should we always call back the application with full buffer sizes at the
requested format, and then keep around any extra data for the next hardware
callback?

This depends on the relation between the driver and callback buffer sizes. What you really want, for maximum reliability, is to keep the CPU load per driver buffer as even as possible. The more it varies, the greater the risk of getting drop-outs, even when the CPU load is well below 100%.

Thus, from a technical POV, it's best to call the application callback exactly once per driver buffer, asking for just the number of samples you need. Without interpolation, that's all there is to it - no intermediate buffers needed. For interpolation, you'll need to keep one or more samples from the previous buffer, but that's all.

Also note that any internal buffering adds to the latency, beyond the latency already defined by the "nominal" buffer sizes. Whenever you ask the application for samples that aren't going into the current driver buffer, you're essentially asking about the "future" - which translates to adding the number of extra samples to the total latency.

//David

On 2006-01-20 14:57:45 +0000, Patrice Mandin wrote:

(In reply to comment # 4)

(In reply to comment # 3)

Should we always call back the application with full buffer sizes at the
requested format, and then keep around any extra data for the next hardware
callback?

This depends on the relation between the driver and callback buffer sizes. What
you really want, for maximum reliability, is to keep the CPU load per driver
buffer as even as possible. The more it varies, the greater the risk of getting
drop-outs, even when the CPU load is well below 100%.

Thus, from a technical POV, it's best to call the application callback exactly
once per driver buffer, asking for just the number of samples you need.

I still wonder why it is such a big issue for you. MOD music players do that since the beginning (replaying samples at 'f1' KHz on a sound device opened at 'f2' KHz). Amiga and Atari ST soundtrackers were not running on GHz machines.

This is also a problem for me on Atari platform (and also on Amiga hardware), because the default hardware device does not support 44.1 or 48 KHz frequencies (even sub-multiples).

Programmers should not assume the audio is 44.1 (or 48), the same way that not everyone has 32 bits video mode, but 15, 16 or 24 sometimes.

On 2006-01-20 16:32:51 +0000, Alex Volkov wrote:

(In reply to comment # 3)

The app requests 4096 sample buffer size at 44.1KHz, driver provides 1024
sample size at 48KHz. We do the following loop:

driver callback:
fill app sound buffer with 4096 samples
consume 940 samples filling 1024 sample sound buffer at 48KHz, leaving 3156

SDL does not guarantee now that the app gets what it calls for (correct me if I am wrong). And I do not think that should change -- I would rather get a different buffer at a rate different from what I asked for, but have control over the buffer provided. Having SDL provide conversion routines capable of arbitrary resampling for my use is wonderful, but I would rather use them consciously than implicitly.

IMHO, the mixer (app-supplied callback) should be responsible for ensuring proper formats and rates, and SDL audio should just get the format from the driver as close as possible. But of course, having arbitrary SDL conversion functions would be nice.
If SDL audio hides the fact that it is interpolating, and it is not interpolating the way I want or need, I can get crappy sound quality. But if I (my app) is responsible for conversions -- I can choose to use SDL conv routines, or choose not to use them and use my own instead (which is what we do in our project right now -- we have own mixer).

As a side note, even a not-so-optimized C cubic interpolation for 44.1KHz stereo is not so slow right now -- it does not even take 1% of 1GHz CPU.

On 2006-01-21 08:12:07 +0000, David Olofson wrote:

(In reply to comment # 5)
[...]

I still wonder why it is such a big issue for you. MOD music players do that
since the beginning (replaying samples at 'f1' KHz on a sound device opened at
'f2' KHz). Amiga and Atari ST soundtrackers were not running on GHz machines.

Actually, it's no big issue IMHO, but supporting arbitrary audio and pixel formats and the like through emulation can be handy for quick porting.

I suppose newer Amigas and STs have more CPU power, but back in the 7.14/8 MHz days, only "nearest sample" resampling was fast enough if you actually wanted to do anything more than sound, at least with more than 4 voices. (The Amiga did 4 voices in hardware, so there was no need to do it in software for the normal 4 channel MODs.)

This is also a problem for me on Atari platform (and also on Amiga hardware),
because the default hardware device does not support 44.1 or 48 KHz
frequencies (even sub-multiples).

Well, the Amiga can do it, but only if you bypass the DMA and feed Paula with the CPU... (I did >44 kHz on my 25 MHz Amiga 3000, and even had about an A500's worth of CPU power left while doing it. :-D )

Programmers should not assume the audio is 44.1 (or 48), the same way that not
everyone has 32 bits video mode, but 15, 16 or 24 sometimes.

Good point, though I personally find the pixel format emulation rather handy for less performance critical work.

Also keep in mind that as long as you don't need custom blitters, you can just use SDL_DisplayFormat*(), and get maximum performance without having to explicitly support any pixel formats at all. As it is now, there is no corresponding solution for audio, as the "use specified format even if it means real time conversion" logic is more or less broken.

On 2006-01-21 08:28:59 +0000, David Olofson wrote:

(In reply to comment # 6)
[...]

SDL does not guarantee now that the app gets what it calls for (correct me if
I am wrong).

Actually, that depends on whether or not you specify a target SDL_AudioSpec for the second argument. If you don't, SDL_OpenAudio() will fail if it cannot provide exactly what you're asking for. There's no need to change this logic. This is just fixing what's already there, for those who want to use it.

IMHO, the mixer (app-supplied callback) should be responsible for ensuring
proper formats and rates, and SDL audio should just get the format from the
driver as close as possible. But of course, having arbitrary SDL conversion
functions would be nice.

Exactly. The conversion is nice to have whenever performance isn't critical enough to implement custom code for 15/16/24/32 pixel formats, audio resampling and whatnot.

If SDL audio hides the fact that it is interpolating, and it is not
interpolating the way I want or need, I can get crappy sound quality. But if I
(my app) is responsible for conversions -- I can choose to use SDL conv
routines, or choose not to use them and use my own instead (which is what we
do in our project right now -- we have own mixer).

As a side note, even a not-so-optimized C cubic interpolation for 44.1KHz
stereo is not so slow right now -- it does not even take 1% of 1GHz CPU.

I have some fixed point cubic interpolators too. Fast, simple and probably good enough as long as you don't need to downsample more than half an octave or so. (Then you'd need a brickwall low pass filter as well.)

BTW, if the conversion ratio is reasonably "nice" (not too many possible fractional sample positions used), one can do away with most of the interpolation code and use a precalculated circular LUT instead. However, I'm not sure it's much point on P-II and better CPUs, as a reasonably fast cubic interpolator is memory bound on those. It might make sense for lower end CPUs, though.

On 2006-01-21 11:58:29 +0000, Sam Lantinga wrote:

David, since you're familiar with audio processing, do you want to take point on this? Set it up so the audio conversion can use arbitrary frequency shifts, and then set up the audio AudioSpec format conversion to use it?

Let's implement a nearest sample version, and the cubic fixed-point whatchamathingie, chosen at compile time.

On 2006-01-21 14:05:18 +0000, Ryan C. Gordon wrote:

The only thing to consider, now that I already started this whirlwind, is that some people might be doing timing off the audio callback (or at least use it to estimate how much latency they can expect to incur, so it might be better to always have the callback fire with a constant size, and keep a small buffer inside SDL to manage the difference).

--ryan.

On 2006-01-21 14:59:12 +0000, Sam Lantinga wrote:

(In reply to comment # 10)

The only thing to consider, now that I already started this whirlwind, is that
some people might be doing timing off the audio callback (or at least use it to
estimate how much latency they can expect to incur, so it might be better to
always have the callback fire with a constant size, and keep a small buffer
inside SDL to manage the difference).

Yes, I think this is a good idea, even if it adds a small amount of latency. The reason being is that we'll only do it if the application requests that SDL do any necessary conversion (NULL in the second audio open parameter), and in that case we want to give the application exactly what they expect - e.g. a fixed size callback buffer.

In the case where the application is smart and wants to handle audio conversion themselves, if our audio conversion routine does good frequency shifting, then they can do whatever they want.

On 2006-01-22 04:44:41 +0000, David Olofson wrote:

(In reply to comment # 9)

David, since you're familiar with audio processing, do you want to take point
on this? Set it up so the audio conversion can use arbitrary frequency shifts,
and then set up the audio AudioSpec format conversion to use it?

I'm not very familiar with the SDL audio code, but I'll look into it.

Let's implement a nearest sample version, and the cubic fixed-point
whatchamathingie, chosen at compile time.

Well, I was thinking it might be nice to have it selectable at run time - but OTOH, if you're writing something that's supposed to scale to Pentiums and weaker CPUs (which is the only place a fixed point cubic interpolator isn't memory bound), you shouldn't rely on SDL's on-the-fly conversions anyway. So, a compile time option is probably fine.

On 2006-01-22 04:55:36 +0000, David Olofson wrote:

(In reply to comment # 10)

The only thing to consider, now that I already started this whirlwind, is that
some people might be doing timing off the audio callback (or at least use it to
estimate how much latency they can expect to incur, so it might be better to
always have the callback fire with a constant size, and keep a small buffer
inside SDL to manage the difference).

The problem is, there are only two ways of doing this: 1) having a separate, timer driven thread do the audio callbacks (otherwise you'd need to make an extra call or skip a call once in a while), or 2) simply rounding the application side buffer size to the nearest integer value and stick with that.

I'd say only the second alternative is viable. The only side effect is that it limits the accuracy - but we're talking about an error of around one percent worst case (minimum "safe" buffer sizes on Mac OS X and Linux/lowlatency), so it's still good enough that you won't be able to tell the difference in normal applications. (A synth may need a slight tune adjustment to play with other instruments, but a synth shouldn't need to use this feature anyway.)

On 2006-01-22 04:58:02 +0000, Ryan C. Gordon wrote:

The problem is, there are only two ways of doing this: 1) having a separate,
timer driven thread do the audio callbacks (otherwise you'd need to make an
extra call or skip a call once in a while), or 2) simply rounding the
application side buffer size to the nearest integer value and stick with that.

With the exception of MacOS Classic (where this runs in a hardware interrupt), all platforms currently use a seperate thread for the audio callback already...Sam, is this accurate?

--ryan.

On 2006-01-22 08:47:37 +0000, David Olofson wrote:

(In reply to comment # 14)

The problem is, there are only two ways of doing this: 1) having a separate,
timer driven thread do the audio callbacks (otherwise you'd need to make an
extra call or skip a call once in a while), or 2) simply rounding the
application side buffer size to the nearest integer value and stick with that.

With the exception of MacOS Classic (where this runs in a hardware interrupt),
all platforms currently use a seperate thread for the audio callback
already...Sam, is this accurate?

That's correct. (At least, it's the only way to do it on Linux, short of application driven polling from the main thread.) However, AFAIK, that thread blocks on the audio device.

The extra thread I'm talking about would have to be driven by a timer in order "fake" a fixed callback rate, so that applications can still derive their timing directly from the callback timing. If you do it from the existing audio thread, there will be N callbacks per actual audio buffer, and SDL will have to make an extra call, or skip a call, every once in a while.

On 2006-01-22 13:52:38 +0000, Sam Lantinga wrote:

That's correct. (At least, it's the only way to do it on Linux, short of
application driven polling from the main thread.) However, AFAIK, that thread
blocks on the audio device.

The extra thread I'm talking about would have to be driven by a timer in order
"fake" a fixed callback rate, so that applications can still derive their
timing directly from the callback timing. If you do it from the existing audio
thread, there will be N callbacks per actual audio buffer, and SDL will have to
make an extra call, or skip a call, every once in a while.

I think this is reasonable. Even if you had a timer thread faking the audio timing you'd still have load related problems unless it was running at real-time priority, which might not even be an option.

Let's consider adding a way to query the current actual audio playback position for 1.3

On 2006-01-25 11:19:11 +0000, David Olofson wrote:

(In reply to comment # 16)
[...]

I think this is reasonable. Even if you had a timer thread faking the audio
timing you'd still have load related problems unless it was running at
real-time priority, which might not even be an option.

Exactly. (Actually, this kind of arrangement tends to generate load problems even with RT priority on an RTOS. It's an unavoidable inherent effect of the design. Basically, just Don't Do That.)

Let's consider adding a way to query the current actual audio playback
position for 1.3

Yes, that would be very handy.

In DT-42, I rely on the callback timing and make a "guess" at how much additional buffering there is. Close enough on three of my machines here (Linux, Win2k and OS X), but that's probably just luck. :-) (The default delay compensation value can be overridden from the command line if need be.)

Anyway, I was looking at the code. Cubic interpolation does a pretty good job of resampling to higher sample rates; it behaves like a rather nice LPF at Nyqvist of the original sample rate, as desired. Free bonus, basically.

However, downsampling isn't all that fun. Though higher order interpolators do a bit better (probably because "skipped" samples still weigh into the output to some extent), one should really use a steep LPF at the input Nyqvist before the resampler to minimize aliazing caused by whatever might be above that frequency.

Then again, this is not much of an issue for 48->44.1, which is probably the most common case. If you're trying to play back 44.1 or 48 kHz audio through something that won't do more than 16 or 22.05 kHz or something, you're probably on a weak CPU, and there's not exactly countless spare cycles to spend on high quality downsampling. (And, the application should of course support the hardware sample rate, avoiding the problem entirely.)

So, do we assume that this stuff is mostly about resampling <insert odd, low sample rate here> to 44.1 or 48 kHz, and accept that downsampling upwards of an octave or more won't sound too great?

On 2006-01-25 11:55:46 +0000, David Olofson wrote:

I'm seeing some issues with the SDL_AudioCVT struct.

Although it's a bit hairy, interpolation can be done in-place, so no major problems there. However, I need somewhere to store two samples from the previous buffer. (Nicest place would be right before the current buffer, to completely avoid special cases around the interpolator.)

One way would be to make this an internal hack, but then this resampler wouldn't work in a real (application created) SDL_AudioCVT. :-/

Dirty hack idea: Abuse the filter callback array... :-) (Use the item after the cubic interpolator as a pointer to private data of some sort.)

On 2006-01-26 03:50:26 +0000, Sam Lantinga wrote:

So, do we assume that this stuff is mostly about resampling <insert odd, low
sample rate here> to 44.1 or 48 kHz, and accept that downsampling upwards of an
octave or more won't sound too great?

Sure, we can always add more options later if we want.

On 2006-01-26 03:57:42 +0000, Sam Lantinga wrote:

(In reply to comment # 18)

I'm seeing some issues with the SDL_AudioCVT struct.

Although it's a bit hairy, interpolation can be done in-place, so no major
problems there. However, I need somewhere to store two samples from the
previous buffer. (Nicest place would be right before the current buffer, to
completely avoid special cases around the interpolator.)

One way would be to make this an internal hack, but then this resampler
wouldn't work in a real (application created) SDL_AudioCVT. :-/

Dirty hack idea: Abuse the filter callback array... :-) (Use the item after the
cubic interpolator as a pointer to private data of some sort.)

Can you do it by assuming the previous two samples are the same as the first sample for resampling purposes? You'd get slight artifacts at the beginning of each buffer, but I doubt they'd be audible.

For 1.3, we can extend the conversion structure to contain an array of void* as parameters for the filter.

On 2006-01-26 05:53:38 +0000, David Olofson wrote:

(In reply to comment # 20)
[...]

Can you do it by assuming the previous two samples are the same as the first
sample for resampling purposes? You'd get slight artifacts at the beginning
of each buffer, but I doubt they'd be audible.

Well, in my experience (synths mostly), the clicking (or rather low frequency buzzing) that this generaten can be at least as annoying as the (constant) artifacts you get without interpolation. High frequency content in the audio stream may mask it to some extent, but that depends completely on the contents.

Extrapolating the missing points might improve things, though. It would make things worse with high frequencies involved, but would reduce clicks in low frequency signals where they're more audible. (What happens is actually that the error moves down a derivate or two.)

I'll try some things and see (hear) how it turns out.

On 2006-01-27 11:23:18 +0000, Ryan C. Gordon wrote:

Setting Sam as "QA Contact" on all bugs (even resolved ones) so he'll definitely be in the loop to any further discussion here about SDL.

--ryan.

On 2006-03-21 00:09:52 +0000, Sam Lantinga wrote:

David, any luck on this? I'd like to include it in SDL 1.2.10, if it's ready.

On 2006-03-21 04:52:33 +0000, David Olofson wrote:

Sorry, no time for (non work related) coding at this point. :-/ We're shipping the first "full" instrument today, but that doesn't mean I'm off the hook just yet...

On 2006-05-07 13:38:03 +0000, Sam Lantinga wrote:

*** Bug 156 has been marked as a duplicate of this bug. ***

On 2006-05-07 17:09:58 +0000, Sam Lantinga wrote:

I'd like to get this fixed for SDL 1.2.10 release, if possible.

On 2006-05-11 04:36:38 +0000, Sam Lantinga wrote:

It looks like David won't have time to work on this for the 1.2.10 release.

On 2007-02-22 02:23:26 +0000, Ryan C. Gordon wrote:

*** Bug 396 has been marked as a duplicate of this bug. ***

On 2007-07-07 16:51:40 +0000, Sam Lantinga wrote:

Updated for implementation in SDL 1.3

On 2009-01-10 19:26:12 +0000, Ryan C. Gordon wrote:

Reassigning bug to myself.

--ryan.

On 2009-02-16 20:40:11 +0000, Sam Lantinga wrote:

Ryan is currently working on this. Ryan, what's left to do?

On 2011-12-28 10:49:32 +0000, wrote:

This Debian bug report seems to be a manifestation of this problem, too:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=460536

On 2013-05-21 00:46:18 +0000, Sam Lantinga wrote:

Ryan, can you take a look and see what's left for 2.0 release?

On 2016-01-03 21:01:30 +0000, Ryan C. Gordon wrote:

*** Bug 2649 has been marked as a duplicate of this bug. ***

On 2016-01-03 21:03:41 +0000, Ryan C. Gordon wrote:

*** Bug 2845 has been marked as a duplicate of this bug. ***

On 2016-12-18 15:21:40 +0000, Vitaly Novichkov wrote:

Created attachment 2652
More accurate resampling

I took care and fixed inaccurate and glitchy resampling myself (with causing extra clicks between chunks and an even bug in the X4 resampler where used a constant 8 instead of actual channels count). I tested it on 8000, 11025, 14700, 22050, 30000, 44100, 48000, 88200, 125121, 132300 and used stereo stream 44100. All sounds are signed 16-bit. Also did tests with the mono and 8bit sources.

The patch made based on the latest revision f6cd81aab88e.

Even that resampler worked, but a sound was glitchy and hat lot of clicks between chunks (because of inaccurate length calculation).

On 2016-12-18 17:12:12 +0000, Vitaly Novichkov wrote:

P.S. Actual patch posted at Bug 3527. This my patch is bit outdated.

On 2017-01-02 03:20:28 +0000, Sam Lantinga wrote:

Pointing at bug 3527 with Vitaly's latest patch

*** This bug has been marked as a duplicate of bug 3527 ***

On 2017-01-06 00:34:48 +0000, Ryan C. Gordon wrote:

Aww man, I was totally going to resolve this ten-year-old bug with the SDL_AudioStream stuff!

--ryan.

On 2017-01-06 06:10:02 +0000, Ryan C. Gordon wrote:

(In reply to Ryan C. Gordon from comment # 39)

Aww man, I was totally going to resolve this ten-year-old bug with the
SDL_AudioStream stuff!

And now I have: https://hg.libsdl.org/SDL/rev/329d6d46fb90

(plus some commits after that to fix up platforms that provide their own audio threads, so they use the new streamer stuff too, and other minor patches.)

--ryan.

On 2017-01-06 06:32:19 +0000, Ryan C. Gordon wrote:

(In reply to Ryan C. Gordon from comment # 40)

(In reply to Ryan C. Gordon from comment # 39)

Aww man, I was totally going to resolve this ten-year-old bug with the
SDL_AudioStream stuff!

(Eleven year old!!)

--ryan.