On Sun, Dec 1, 2013 at 6:48 AM, Fons Adriaensen <fons(a)linuxaudio.org> wrote: linus has always insisted (rightly or wrongly) that user-space code should assume omnipotence and beneficence on the part of the kernel with respect to I/O. there are plenty of people who don't agree with him, hence things like the madvise() call. but i think that it is a pretty good bet for most cases. in ardour, the purpose of the disk i/o thread (the "butler") is primarily to organize buffering within user space based on observed (but very occasional) delays in I/O. it is not attempting to do better than the kernel in handling read-ahead and in fact for quite a lot of situations, ardour really does rely on the kernel's own buffering to do a good job. we only buffer "more" because of the delays that have been seen over the years, which suggest that is expedient to not rely on data being available from read() within any reliable time frame less than a few seconds. several years ago, linus told me in no uncertain terms not to use mmap for disk i/o. his perspective is that even though it works, the kernel is not developed or optimized with this in mind, and that normal streaming i/o interfaces like read/write will always be the right path. if you use stdio, you add an additional, hidden layer of buffering that really provides very few benefits given that you will likely be reading/writing fairly large chunks already. read and write are certainly my preference, although to handle the possibility of multithreaded access to the same file descriptor, pread and pwrite are attractive. the kernel provides internal implementations of this, as does the firmware on almost any storage device. probably not worth reimplementing (especially given that you don't have any special knowledge about the device) seems application-centric. some cases would need this some would not. i would still try to add support for this format to libsndfile, so as to gain all the benefits it offers from an application perspective.

On Sun, Dec 1, 2013 at 11:49 AM, Fons Adriaensen <fons(a)linuxaudio.org>wrote;wrote: basically, all files are kept open all the time. in reality, because some systems impose limits on the number of open files, we keep a LRU cache for file descriptors. most users never will run into a situation where the cache is actively used. we read ahead by an amount decided by the user (default is 5 seconds). we write in chunks of 256kB because very old experiments suggested that on ext3 we got the best throughput with that (this predated our use of libsndfile, however. ardour does not attempt to optimize this. it will call read() with the file descriptor corresponding to A 4 times, the one corresponding to B 3 times, the one corresponding to C 4 times and the one corresponding to D once. each time, it will read the same data each time. this is what i mean by relying on the kernel's own buffering. those second reads typically get an estimated bandwidth on the order of GBytes/second (i.e. basically just memcpy from the kernel buffer cache into user space, because the kernel already has the data cached). you could construct arbitrary scenarios where the kernel's buffering strategy is wrong, but i'm not interested in the work involved in designing a better strategy. there were papers i read in the early 2000's from the late 1990's that outlined such designs (and filesystems for audio) - AFAICT, all of this work has basically been made irrelevant by contemporary hardware and OS performance. when you can stream 100 32 bit float mono interleaved tracks from a single (cheap) SATA disk without real issues, i frankly don't see the point in design work to squeeze a few more tracks out of the same setup.