That, as they say, is the $64,000 question!  :)

In addition to the advice already offered by Paul, a lot depends on how important latency is to your specific application.  You mentioned that the audio was coming off of a radio receiver?  Modern DAB transmission air chains already typically incur several *seconds* of latency, so if that’s your use case, then I’d go ahead and buffer up a couple of seconds more on the receive end.  Cheap insurance.  OTOH, if minimizing latency is important, then the answer to your question is “buffer as little as you can get away with without incurring xruns”.  The only way to determine where that sweet spot is located is by testing your code on its target hardware.

Cheers!