The point was that NOT cancelling introduces extra latency unnecessarily, and short requests introduce CPU overhead. I gave the NFS example to preempt the "hard disks are really fast" objection that some people are always tempted to make... For the given example, if the NFS server can stream at even 110% of the playback speed without jitter, you can still achieve glitch-free, continuous playback with as low a *skip* latency as the CPU (and chosen request size) permit. *If* you can cancel active requests, that is -- otherwise you need to double that latency. And yes, I actually do stream from a fileserver (over NFS) myself quite often. The problem of a SNDFILE becoming invalid after a cancelled I/O. Yes, there is a problem -- valgrind reports nasty memory errors (and eventually I get a segfault) if I don't re-open the SNDFILE. [ If you meant I should test the sf_seek() trick first -- I doubt that testing a few sf_seek()s on a few FLAC files will reliably establish whether sf_seek() really does reset all encoder and decoder structure under all circumstances. Especially since you said you weren't sure yourself. ] -- Dan

This is what I was trying to do via pthread_cancel + sf_open_fd. It seems to work with FLAC. However, there could be memory leaks if any decoder / encoder function (in sndfile or one of the external libs) allocates temporary memory that it expects to free before returning. Do you know if this might be the case (e.g. for the native formats, libFLAC, libvorbis)? I may need to fall back on the vio stuff. How does a sf_vio_read or sf_vio_write signal I/O errors? How does a vio_read signal EOF? The API docs don't seem to say anything. -- Dan

It seems to works for FLAC, but I can't trust it either. Besides memory leaks, mutexes might remain locked, and indirectly deadlock sf_free(). Well, forget my cancelling strategy -- errors could occur even if one writes a trivial layer (vio_read == read). An I/O API *must* specify error handling. This is not some exotic use case... if (psf->virtual_io) return psf->vio.read ... (i.e. no processing), while the non-virtual branch returns the partial count if an error ever occurs. So I guess I should return some short count on error. Anyway, for cancelling requests EOF works -- indeed as long as one can "get out of the sf_read jail free" anything works. -- Dan

Dan Muresan wrote: Maybe not on FLAC. I would suggest that you write a small standalone test program to test this and if you don't get what you think you should, send me the test program and an explanation of what you think you should get. This is not possible on all file types, especially not on compressed formats like FLAC. It does however work on PCM data for most file formats. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: You need to supply me with a small test program and if necessary this broken FLAC file. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: I never much liked the VIO layer. It was only ever added because a large number of people requested it. I think its fragile and it exposes too much of libsndfile internals to the user. I would be surprised if reset-by-seek didn't work on standard WAV/AIFF etc files with PCM data accessed via the standard sf_read/write* functions. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: Yes, but it means that I can't decide to replace all uses of write() with pwrite() for instance. At one stage I though this was a good idea. This lack of clarity is one of the things I don't like. More that bugs or inconsistency in the user provided VIO functions look like libsndfile bugs. You shouldn't have too. Anything that you can do with 16 bit PCM WAV files that you can't do with FLAC (and libsndfile doesn't say there is an error) is probably a bug. It should for all the standard file formats. If it doesn't for FLAC then thats a bug that I would try to fix or make sure that libsndfile notifies the client program that something has gone wrong. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: Whaaat? You're kidding me right? I haven't measured it, but my educated guess is that if you're reading 16384 frames at a time from a FLAC file on current hardware, then the difference between reading st_blksize sized blocks and non-st_blksize sized blocks will be absolutley swamped, by disk latencies, cache latencies, scheduling latencies and file decoding overhead. You might want to google the term "premature optimisation". Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: I think you missed the point. The point is that if you set out to measure the difference (in CPU load or in latency) between reading st_blksize size blocks and st_blocksize - 4 blocks at the libsndfile API level, your measurements would be so swamped with noise from other factors that any differences would be statistically irrelevant. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: Regardless of whether you choose to measure latency or CPU load if you vary the st_blksize as specified in the previous email, you will not be able distinguish between the two values of st_blksize due to the influence of other factors. Don't you think it would be better to measure rather than rely on standards and common practice, both of which may be wrong when it comes to things like performance issues? The behavior of fread() can be dubious for files accessed via NFS with respect to incomplete reads (ie EAGAIN). The only solution to this is to use read(). Breaking what rules? You provide concrete proof that doing block sized reads makes any positive performance improvement and I'll implement block sized reads and buffering. Until you can show concrete proof I consider this issue closed. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

Dan Muresan wrote: Its late, I'm tied and I'm frustrated at your ability to miss the point every time. Let me fix my last statement: Regardless of whether you choose to measure latency or CPU load if you test two read sizes (st_blksize and st_blksize - 4) you will not be able distinguish between the two read sizes due to the influence of other factors. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

On Thu, Jul 7, 2011 at 9:38 AM, Dan Muresan <danmbox(a)gmail.com> wrote: [ ... ] Dan - I'm really confused by what you're getting at, or trying to get to. It seems as though you think there is a difference in the throughput when you read from a filesystem using st_blksize or some other size close to it, and that you either have or could measure it. Erik is saying that he doesn't believe that you could measure the difference. The only empirical data I have comes from years ago (maybe 8-9 years ago) when I measured this with ext2 and possibly 2 other linux filesystems, and found that the best throughput was obtained using values very, very far removed from st_blksize. This is why ardour tries to write in chunks of 256kB, although in truth it would be better if we measured this again on ext3 and newer FS's, and if possible provided a run-time measurement.

On Thu, Jul 7, 2011 at 10:28 AM, Dan Muresan <danmbox(a)gmail.com> wrote: I don't consider man pages definitive guides for *anything*. Not even for system calls that are part of POSIX. If you want to measure this, go ahead. If what is described on the man page matches what you find, that's great, and it will be like (as a rough guess) 90% of the rest of the OS API. But if it doesn't, don't be suprised. One of the things about open source is that if you actually spend time reading (in this case, kernel) source, you rapidly come to discount man pages and other documentation as sources of definitive information. This might be a bad development (there are some good arguments to this effect), but it doesn't change the fact that it happens. However, I think you're missing Erik's point. He's not claiming that there isn't a "rule" (your term, though I think its inappropriate) for setting the i/o size based on st_blksize. He's saying that if you break the "rule" in the way that is being discussed, you will find it hard to see any effect of doing so. He may have also explained why this is likely to be the case in the future too (or I've just imagined that part). So the situation is not: * do it this way, or it will break its more like: 1) there's a reason to use value A 2) there's a different reason to use value B 3) there's no measurable difference in performance between A and B, and thus the choice between 1+2 must be made on some other grounds Future-proof and standards surely have almost no relationshiop to each other. The future is where today's standards are changed, or discarded. The ghosts of thousand dead standards will back me up on this one. Variations of less than about 500 bytes made no statistically significant difference to measured throughput. Sure, but its because we care about performance, not standards or "rules". And I measured it first.

On Tue, Jun 14, 2011 at 01:01:09PM +0300, Dan Muresan wrote: 'Work around it' coild mean something entirely different : not trying to cancel anything, by using a system that doesn't require it. More below. It's probably a Good Thing (TM) that libsndfile doesn't do its own buffering. First, the file system will do buffering, and if you're playing a sound file probably decide it should read ahead. Second, any application that wants to use audio data in real time will need to do its own buffering, typically a few seconds. How exactly this is organised will be application dependent. Having a third layer in between this and the system is not going to help. If you support things like loops, random locates, shutthling, scrubbing, etc. the buffer management logic required to support all of this in an efficient way may become complicated. But only the application can do it. One way to organise the buffer is to divide it in fragments of say 1/4 second, and have the start offset of each one quantised to the same value. The syscall overhead becomes trivial for such a lenght. When you need new fragments, you send commands to the reader thread to read them. If the requirements change before they are read, you can still cancel all but one of them (you cancel the requests, not the actual read() call). The one you can't cancel is no big deal, just let it happen. It will take little time compared to what has to be done anyway. Ciao, -- FA

On Tue, Jun 14, 2011 at 02:42:18PM +0300, Dan Muresan wrote: Not only copying (which is cheap anyway), but also logic. Your application layer cache knows what you want to achieve, will have some ad-hoc strategies and try to do the right thing. The intermediate one doesn't have that and may well work against you. The same problem exists with the system level buffering, but you can't do much about that. I don't know what your app is doing, so I'll assume it's some sort of player. Now if you relocate, you send the commands to read the data at the new position to your reader thread. Assume your buffer is 2 seconds, so that's 8 commands to read 1/4 of second. You can't safely start playback again unless you have at least a second or so buffered. No assume you have a new relocate before that time. Again you send the commands to read 8 blocks of 1/4 second. There is some logic in the app that makes these cancel the previous ones that have not yet started. So you end up with 1 you can't cancel against 8 that have to be done anyway. That's not big loss. One of my players works this way. From a user point of view a relocate happens almost instantly. If the read bandwidth is just above what is required for continuous streaming, then very probably you can't support this sort of thing without extra delays. But even in this case they don't need to be very big. In the example above one extra fragment is read (the one you can't cancel) compared to the four or so you need to have done anyway before you can resume playing safely. So it takes as most 25% more time. If the reads happen on an NFS volume then even if the cancel would work on the client side that doesn't imply that the data won't be transmitted by the server anyway. So nothing would be gained by cancelling. Ciao, -- FA