13 Jul
2006
13 Jul
'06
6:46 p.m.
A thought occured to me recently...
If I am writing an application which needs to stream a large wav file,
I am having to write something which reserves some memory, and loads
pieces of the wav file from disk on request. Say I need to be able to
jump around the file a bit, I would have to detect when the piece is
not available and load it in as appropriate.
... which I realized is exactly what the OS VM paging system does.
So, has anyone tried using memory-mapped files for streaming audio
data? (obviously, I mean, in a non-realtime thread, using a buffer).
Or would this be totally inefficient?
I was thinking it could really simplify programming, just directly
accessing parts of the wav file as needed, and letting the OS load it
up into physical memory when it wants to.
Just curious.
Steve
13 Jul
13 Jul
6:56 p.m.
If you need to stream a file, mmap'ed variant will eat memory up to
file size. Given large enough file, it will eat all you memory and
then will begin to page out unused portions of the file.
Of course, details on when and where will vary depending on VM-system
implementation.
The thing is, you really need only a small portion of a file. Memory
mapping is useful if you need all data to be (virtually) present in
RAM.
Regards,
Dmitry.
7:15 p.m.
On Fri, 2006-07-14 at 00:56 +0400, Dmitry Baikov wrote:
If you need to stream a file, mmap'ed variant will
eat memory up to
file size. Given large enough file, it will eat all you memory and
then will begin to page out unused portions of the file.
Of course, details on when and where will vary depending on VM-system
implementation.
The thing is, you really need only a small portion of a file. Memory
mapping is useful if you need all data to be (virtually) present in
RAM.
moreover, in 2001-2002, it was actually slower to do i/o this way. when
asked about linus said "i know and i intend to keep it that
way" (paraphrasing).
--p
11:31 p.m.
Stephen Sinclair wrote:
The other good thing about this is that once you give up
mem-mapping you can just use libsndfile and have a bunch of
different file formats other than WAV.
Erik
--
+-----------------------------------------------------------+
Erik de Castro Lopo
+-----------------------------------------------------------+
GPLG
GPLGPLGP
GPLGPLGPLGP
GPLGP
GPL MICROSOFT
GPLGP
GPLGPLGPLGP
GPLGPLGPL
GPLGPL
asked about
linus said "i know and i intend to keep it that
way" (paraphrasing).
ah.
i take it it's not a good idea then.. ;-)
thanks for the answers, they were informative. 
9:06 p.m.
On Thu, 13 Jul, 2006 at 04:45PM -0400, Stephen Sinclair spake thus:
A thought occured to me recently...
If I am writing an application which needs to stream a large wav file,
I am having to write something which reserves some memory, and loads
pieces of the wav file from disk on request. Say I need to be able to
jump around the file a bit, I would have to detect when the piece is
not available and load it in as appropriate.
... which I realized is exactly what the OS VM paging system does.
So, has anyone tried using memory-mapped files for streaming audio
data? (obviously, I mean, in a non-realtime thread, using a buffer).
Or would this be totally inefficient?
I was thinking it could really simplify programming, just directly
accessing parts of the wav file as needed, and letting the OS load it
up into physical memory when it wants to.
It's perfectly sensible. I do it a lot, because it's easy. The
problem is having to load the whole thing into memory before you
start, which makes things alow to start. If you're playing linearly,
to solve this, you need can load it in chunks and start playing the
first chunk straight away.
Just curious.
Steve
14 Jul
14 Jul
8:12 a.m.
On Fri, Jul 14, 2006 at 12:06:23 +0100, james(a)dis-dot-dat.net wrote:
On Thu, 13 Jul, 2006 at 04:45PM -0400, Stephen
Sinclair spake thus:
mmap-ing doesn't epxlicitly load the whole file, just bits as you
request them. I wrote an app on linux that accesses large files (1-4 GB)
this way recently, and the performance was certainly no worse than
buffered read()s.
It is very convienient, but there are some gotchas, especailly on 32bit
machines where you will run out of address space really quickly. Also, my
feeling is that linux is a bit conservative about how much ram it will use
to map the file, though there is a hinting mechanism you can use to say
what you want that mmaped region for. It's madvise(2) on Linux and BSD.
I'd second what Erik said though, for audio file i/o, use a library. The
ammount of i/o is really tiny, so overoptimising it is silly.
- Steve
A thought occured to me recently...
If I am writing an application which needs to stream a large wav file,
I am having to write something which reserves some memory, and loads
pieces of the wav file from disk on request. Say I need to be able to
jump around the file a bit, I would have to detect when the piece is
not available and load it in as appropriate.
... which I realized is exactly what the OS VM paging system does.
So, has anyone tried using memory-mapped files for streaming audio
data? (obviously, I mean, in a non-realtime thread, using a buffer).
Or would this be totally inefficient?
I was thinking it could really simplify programming, just directly
accessing parts of the wav file as needed, and letting the OS load it
up into physical memory when it wants to.
It's perfectly sensible. I do it a lot, because it's easy. The
problem is having to load the whole thing into memory before you
start, which makes things alow to start. If you're playing linearly,
to solve this, you need can load it in chunks and start playing the
first chunk straight away.
6766
days inactive
6767
days old
linux-audio-dev@lists.linuxaudio.org
6 comments
6 participants
participants (6)
-
Dmitry Baikov -
Erik de Castro Lopo -
james＠dis-dot-dat.net -
Paul Davis -
Stephen Sinclair -
Steve Harris