Re: [LAD] Pipes vs. Message Queues

Very interesting. You might be running into some basic scheduler weirdness here though and not something inherently wrong with the POSIX queues. I ran your code here a few times in some different configurations. The results with 1M messages had wild variance with SCHED_FIFO, sometimes 2s, 4s, 6s, etc. Not reliable - although without rescheduling they did seem more consistent. These below are with 10M to give longer run times: a. no SCHED_FIFO, 10M cycles [nicky@fidelispc] /tmp [65] cc ipc.c -lrt [nicky@fidelispc] /tmp [66] ./a.out 4096 10000000 Sending a 4096 byte message 10000000 times. Pipe recv time: 23.220948 Pipe send time: 23.220820 Queue recv time: 13.949289 Queue send time: 13.949226 b. SCHED_FIFO, again 10M cycles. [nicky@fidelispc] /tmp [69] cc ipc.c -lrt -DSET_RT_SCHED=1 [nicky@fidelispc] /tmp [70] ./a.out 4096 10000000 Sending a 4096 byte message 10000000 times. Pipe send time: 34.514288 Pipe recv time: 34.514404 Queue send time: 19.004525 Queue recv time: 19.004427 This was on a dual core laptop, 2.2GHz, no speed stepping, was also watching the top whilst running this. Without FIFO the system CPU spreads across both cores, they both run up towards 100% load for both IPC methods. With SCHED_FIFO/pipe the load does not distribute - I get 94% system load on a single CPU whilst running through the loop. The POSIX code did not show this effect. Odd results so had a look at vmstat rather than top, this gives some indication: No rescheduling: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 1 0 0 6506924 312036 595456 0 0 0 196 750 1569 5 94 0 0 1 0 0 6489188 312036 613344 0 0 0 0 961 25508 8 88 3 0 1 0 0 6488724 312036 613536 0 0 0 0 991 21070 6 92 2 0 1 0 0 6488812 312036 613552 0 0 0 0 697 1446 5 94 1 0 SCHED_FIFO pipe(): procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 2 0 0 6516912 311372 586924 0 0 0 108 569 435180 5 46 44 4 2 0 0 6516272 311372 586972 0 0 0 0 556 436042 3 47 50 0 1 0 0 6516608 311372 586972 0 0 0 0 548 436482 6 46 48 0 1 0 0 6516928 311372 586924 0 0 0 0 563 435930 2 51 48 0 Ouch. Almost 100 times the number of context switches. Is the whole kernel bound up in a single thread doing process context switches? SCHED_FIFO message queues - generally lower, far more variance: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 1 0 0 6510328 316116 587048 0 0 0 0 427 142445 2 83 15 0 1 1 0 6509736 316120 587044 0 0 0 44 440 795 3 97 1 0 1 0 0 6509900 316132 587052 0 0 0 64 439 281037 9 69 22 0 1 0 0 6509436 316132 587048 0 0 0 0 437 796 3 95 2 0 1 0 0 6509868 316160 587020 0 0 0 164 452 151290 5 81 15 0 Vanilla kernel Linux fidelispc 2.6.32-35-generic #78-Ubuntu SMP Tue Oct 11 16:11:24 UTC 2011 x86_64 GNU/Linux Also tested with unbalanced priorities in the sender and receiver, and with only prioritising one of them, pretty much the same as 90/90. Not sure if that helps any. I have another system with a single core, might try it out there later since my results were very different. Regards, nick. "we have to make sure the old choice [Windows] doesn't disappear”. Jim Wong, president of IT products, Acer > > $ ./ipc 4096 1000000 > Sending a 4096 byte message 1000000 times. > Pipe recv time: 6.881104 > Pipe send time: 6.880998 > Queue send time: 1.938512 > Queue recv time: 1.938581 > > Whoah. Which made me wonder what happens with realtime priority > (SCHED_FIFO priority 90): > > $ ./ipc 4096 1000000 > Sending a 4096 byte message 1000000 times. > Pipe send time: 5.195232 > Pipe recv time: 5.195475 > Queue send time: 5.224862 > Queue recv time: 5.224987 > > Pipes get a bit faster, and POSIX message queues get dramatically > slower. Interesting. > > I am opening the queues as blocking here, and both sender and receiver > are at the same priority, and aggressively pumping the queue as fast as > they can, so there is a lot of competition and this is not an especially > good model of any reality we care about, but it's interesting > nonetheless. > > The first result really has me thinking how much Jack would benefit from > using message queues instead of pipes and sockets. It looks like > there's definitely potential here... I might try to write a more > scientific benchmark that better emulates the case Jack would care about > and measures wakeup latency, unless somebody beats me to it. That test > could have the shm + wakeup pattern Jack actually uses and benchmark it > vs. actually firing buffer payload over message queues... > > But I should be doing more pragmatic things, so here's this for now :) > > Program is here: http://drobilla.net/files/ipc.c > > Cheers, > > -dr > > _______________________________________________ > Linux-audio-dev mailing list > Linux-audio-dev(a)lists.linuxaudio.org > http://lists.linuxaudio.org/listinfo/linux-audio-dev