[LAD] Pipes vs. Message Queues

[Nick Copeland]

> From: d at drobilla.net
> To: linux-audio-dev at lists.linuxaudio.org
> Date: Thu, 24 Nov 2011 19:10:26 -0500
> Subject: [LAD] Pipes vs. Message Queues
> 
> I got curious, so I bashed out a quick program to benchmark pipes vs
> POSIX message queues.  It just pumps a bunch of messages through the
> pipe/queue in a tight loop.  The results were interesting:

Very interesting.

You might be running into some basic scheduler weirdness here though
and not something inherently wrong with the POSIX queues. I ran your
code here a few times in some different configurations. The results with 1M 
messages had wild variance with SCHED_FIFO, sometimes 2s, 4s, 6s, etc. 
Not reliable - although without rescheduling they did seem more consistent. 

These below are with 10M to give longer run times:

a. no SCHED_FIFO, 10M cycles

[nicky at fidelispc] /tmp [65] cc ipc.c -lrt 
[nicky at fidelispc] /tmp [66] ./a.out 4096 10000000
Sending a 4096 byte message 10000000 times.
Pipe recv time:  23.220948
Pipe send time:  23.220820
Queue recv time: 13.949289
Queue send time: 13.949226

b. SCHED_FIFO, again 10M cycles.

[nicky at fidelispc] /tmp [69] cc ipc.c -lrt -DSET_RT_SCHED=1
[nicky at fidelispc] /tmp [70] ./a.out 4096 10000000
Sending a 4096 byte message 10000000 times.
Pipe send time:  34.514288
Pipe recv time:  34.514404
Queue send time: 19.004525
Queue recv time: 19.004427

This was on a dual core laptop, 2.2GHz, no speed stepping, was also 
watching the top whilst running this. 

Without FIFO the system CPU spreads across both cores, they both run up

towards 100% load for both IPC methods.


With SCHED_FIFO/pipe the load does not distribute - I get 94% system load 
on a single CPU whilst running through the loop. 

The POSIX code did not show this effect.

Odd results so had a look at vmstat rather than top, this gives some indication:

No rescheduling:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 1  0      0 6506924 312036 595456    0    0     0   196  750 1569  5 94  0  0
 1  0      0 6489188 312036 613344    0    0     0     0  961 25508  8 88  3  0
 1  0      0 6488724 312036 613536    0    0     0     0  991 21070  6 92  2  0
 1  0      0 6488812 312036 613552    0    0     0     0  697 1446  5 94  1  0

SCHED_FIFO pipe():
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 2  0      0 6516912 311372 586924    0    0     0   108  569 435180  5 46 44  4
 2  0      0 6516272 311372 586972    0    0     0     0  556 436042  3 47 50  0
 1  0      0 6516608 311372 586972    0    0     0     0  548 436482  6 46 48  0
 1  0      0 6516928 311372 586924    0    0     0     0  563 435930  2 51 48  0

Ouch. Almost 100 times the number of context switches. Is the whole kernel
bound up in a single thread doing process context switches?

SCHED_FIFO message queues - generally lower, far more variance:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 1  0      0 6510328 316116 587048    0    0     0     0  427 142445  2 83 15  0
 1  1      0 6509736 316120 587044    0    0     0    44  440  795  3 97  1  0
 1  0      0 6509900 316132 587052    0    0     0    64  439 281037  9 69 22  0
 1  0      0 6509436 316132 587048    0    0     0     0  437  796  3 95  2  0
 1  0      0 6509868 316160 587020    0    0     0   164  452 151290  5 81 15  0

Vanilla kernel Linux fidelispc 2.6.32-35-generic #78-Ubuntu SMP Tue 
Oct 11 16:11:24 UTC 2011 x86_64 GNU/Linux

Also tested with unbalanced priorities in the sender and receiver, and with only
prioritising one of them, pretty much the same as 90/90.

Not sure if that helps any. I have another system with a single core, might try it
out there later since my results were very different.

Regards, nick.

"we have to make sure the old choice [Windows] doesn't disappear”.
Jim Wong, president of IT products, Acer


> From: d at drobilla.net
> To: linux-audio-dev at lists.linuxaudio.org
> Date: Thu, 24 Nov 2011 19:10:26 -0500
> Subject: [LAD] Pipes vs. Message Queues
> 
> I got curious, so I bashed out a quick program to benchmark pipes vs
> POSIX message queues.  It just pumps a bunch of messages through the
> pipe/queue in a tight loop.  The results were interesting:
> 
> $ ./ipc 4096 1000000
> Sending a 4096 byte message 1000000 times.
> Pipe recv time:  6.881104
> Pipe send time:  6.880998
> Queue send time: 1.938512
> Queue recv time: 1.938581
> 
> Whoah.  Which made me wonder what happens with realtime priority
> (SCHED_FIFO priority 90):
> 
> $ ./ipc 4096 1000000
> Sending a 4096 byte message 1000000 times.
> Pipe send time:  5.195232
> Pipe recv time:  5.195475
> Queue send time: 5.224862
> Queue recv time: 5.224987
> 
> Pipes get a bit faster, and POSIX message queues get dramatically
> slower.  Interesting.
> 
> I am opening the queues as blocking here, and both sender and receiver
> are at the same priority, and aggressively pumping the queue as fast as
> they can, so there is a lot of competition and this is not an especially
> good model of any reality we care about, but it's interesting
> nonetheless.
> 
> The first result really has me thinking how much Jack would benefit from
> using message queues instead of pipes and sockets.  It looks like
> there's definitely potential here... I might try to write a more
> scientific benchmark that better emulates the case Jack would care about
> and measures wakeup latency, unless somebody beats me to it.  That test
> could have the shm + wakeup pattern Jack actually uses and benchmark it
> vs. actually firing buffer payload over message queues...
> 
> But I should be doing more pragmatic things, so here's this for now :)
> 
> Program is here: http://drobilla.net/files/ipc.c
> 
> Cheers,
> 
> -dr
> 
> _______________________________________________
> Linux-audio-dev mailing list
> Linux-audio-dev at lists.linuxaudio.org
> http://lists.linuxaudio.org/listinfo/linux-audio-dev
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxaudio.org/pipermail/linux-audio-dev/attachments/20111125/39b8e0b8/attachment.html>