25 Nov
2011
25 Nov
'11
1:10 a.m.
I got curious, so I bashed out a quick program to benchmark pipes vs
POSIX message queues. It just pumps a bunch of messages through the
pipe/queue in a tight loop. The results were interesting:
$ ./ipc 4096 1000000
Sending a 4096 byte message 1000000 times.
Pipe recv time: 6.881104
Pipe send time: 6.880998
Queue send time: 1.938512
Queue recv time: 1.938581
Whoah. Which made me wonder what happens with realtime priority
(SCHED_FIFO priority 90):
$ ./ipc 4096 1000000
Sending a 4096 byte message 1000000 times.
Pipe send time: 5.195232
Pipe recv time: 5.195475
Queue send time: 5.224862
Queue recv time: 5.224987
Pipes get a bit faster, and POSIX message queues get dramatically
slower. Interesting.
I am opening the queues as blocking here, and both sender and receiver
are at the same priority, and aggressively pumping the queue as fast as
they can, so there is a lot of competition and this is not an especially
good model of any reality we care about, but it's interesting
nonetheless.
The first result really has me thinking how much Jack would benefit from
using message queues instead of pipes and sockets. It looks like
there's definitely potential here... I might try to write a more
scientific benchmark that better emulates the case Jack would care about
and measures wakeup latency, unless somebody beats me to it. That test
could have the shm + wakeup pattern Jack actually uses and benchmark it
vs. actually firing buffer payload over message queues...
But I should be doing more pragmatic things, so here's this for now :)
Program is here: http://drobilla.net/files/ipc.c
Cheers,
-dr
25 Nov
25 Nov
12:07 p.m.
From: d(a)drobilla.net
To: linux-audio-dev(a)lists.linuxaudio.org
Date: Thu, 24 Nov 2011 19:10:26 -0500
Subject: [LAD] Pipes vs. Message Queues
I got curious, so I bashed out a quick program to benchmark pipes vs
POSIX message queues. It just pumps a bunch of messages through the
pipe/queue in a tight loop. The results were interesting:
Very interesting.
You might be running into some basic scheduler weirdness here though
and not something inherently wrong with the POSIX queues. I ran your
code here a few times in some different configurations. The results with 1M
messages had wild variance with SCHED_FIFO, sometimes 2s, 4s, 6s, etc.
Not reliable - although without rescheduling they did seem more consistent.
These below are with 10M to give longer run times:
a. no SCHED_FIFO, 10M cycles
[nicky@fidelispc] /tmp [65] cc ipc.c -lrt
[nicky@fidelispc] /tmp [66] ./a.out 4096 10000000
Sending a 4096 byte message 10000000 times.
Pipe recv time: 23.220948
Pipe send time: 23.220820
Queue recv time: 13.949289
Queue send time: 13.949226
b. SCHED_FIFO, again 10M cycles.
[nicky@fidelispc] /tmp [69] cc ipc.c -lrt -DSET_RT_SCHED=1
[nicky@fidelispc] /tmp [70] ./a.out 4096 10000000
Sending a 4096 byte message 10000000 times.
Pipe send time: 34.514288
Pipe recv time: 34.514404
Queue send time: 19.004525
Queue recv time: 19.004427
This was on a dual core laptop, 2.2GHz, no speed stepping, was also
watching the top whilst running this.
Without FIFO the system CPU spreads across both cores, they both run up
towards 100% load for both IPC methods.
With SCHED_FIFO/pipe the load does not distribute - I get 94% system load
on a single CPU whilst running through the loop.
The POSIX code did not show this effect.
Odd results so had a look at vmstat rather than top, this gives some indication:
No rescheduling:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
1 0 0 6506924 312036 595456 0 0 0 196 750 1569 5 94 0 0
1 0 0 6489188 312036 613344 0 0 0 0 961 25508 8 88 3 0
1 0 0 6488724 312036 613536 0 0 0 0 991 21070 6 92 2 0
1 0 0 6488812 312036 613552 0 0 0 0 697 1446 5 94 1 0
SCHED_FIFO pipe():
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
2 0 0 6516912 311372 586924 0 0 0 108 569 435180 5 46 44 4
2 0 0 6516272 311372 586972 0 0 0 0 556 436042 3 47 50 0
1 0 0 6516608 311372 586972 0 0 0 0 548 436482 6 46 48 0
1 0 0 6516928 311372 586924 0 0 0 0 563 435930 2 51 48 0
Ouch. Almost 100 times the number of context switches. Is the whole kernel
bound up in a single thread doing process context switches?
SCHED_FIFO message queues - generally lower, far more variance:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
1 0 0 6510328 316116 587048 0 0 0 0 427 142445 2 83 15 0
1 1 0 6509736 316120 587044 0 0 0 44 440 795 3 97 1 0
1 0 0 6509900 316132 587052 0 0 0 64 439 281037 9 69 22 0
1 0 0 6509436 316132 587048 0 0 0 0 437 796 3 95 2 0
1 0 0 6509868 316160 587020 0 0 0 164 452 151290 5 81 15 0
Vanilla kernel Linux fidelispc 2.6.32-35-generic #78-Ubuntu SMP Tue
Oct 11 16:11:24 UTC 2011 x86_64 GNU/Linux
Also tested with unbalanced priorities in the sender and receiver, and with only
prioritising one of them, pretty much the same as 90/90.
Not sure if that helps any. I have another system with a single core, might try it
out there later since my results were very different.
Regards, nick.
"we have to make sure the old choice [Windows] doesn't disappear”.
Jim Wong, president of IT products, Acer
From: d(a)drobilla.net
To: linux-audio-dev(a)lists.linuxaudio.org
Date: Thu, 24 Nov 2011 19:10:26 -0500
Subject: [LAD] Pipes vs. Message Queues
I got curious, so I bashed out a quick program to benchmark pipes vs
POSIX message queues. It just pumps a bunch of messages through the
pipe/queue in a tight loop. The results were interesting:
>
> $ ./ipc 4096 1000000
> Sending a 4096 byte message 1000000 times.
> Pipe recv time: 6.881104
> Pipe send time: 6.880998
> Queue send time: 1.938512
> Queue recv time: 1.938581
>
> Whoah. Which made me wonder what happens with realtime priority
> (SCHED_FIFO priority 90):
>
> $ ./ipc 4096 1000000
> Sending a 4096 byte message 1000000 times.
> Pipe send time: 5.195232
> Pipe recv time: 5.195475
> Queue send time: 5.224862
> Queue recv time: 5.224987
>
> Pipes get a bit faster, and POSIX message queues get dramatically
> slower. Interesting.
>
> I am opening the queues as blocking here, and both sender and receiver
> are at the same priority, and aggressively pumping the queue as fast as
> they can, so there is a lot of competition and this is not an especially
> good model of any reality we care about, but it's interesting
> nonetheless.
>
> The first result really has me thinking how much Jack would benefit from
> using message queues instead of pipes and sockets. It looks like
> there's definitely potential here... I might try to write a more
> scientific benchmark that better emulates the case Jack would care about
> and measures wakeup latency, unless somebody beats me to it. That test
> could have the shm + wakeup pattern Jack actually uses and benchmark it
> vs. actually firing buffer payload over message queues...
>
> But I should be doing more pragmatic things, so here's this for now :)
>
> Program is here: http://drobilla.net/files/ipc.c
>
> Cheers,
>
> -dr
>
> _______________________________________________
> Linux-audio-dev mailing list
> Linux-audio-dev(a)lists.linuxaudio.org
> http://lists.linuxaudio.org/listinfo/linux-audio-dev
1:33 p.m.
Nick Copeland wrote:
> I got curious, so I bashed out a quick program to
benchmark pipes vs
> POSIX message queues. It just pumps a bunch of messages through the
> pipe/queue in a tight loop.
This benchmark measures data transfer bandwidth. If increasing that
were your goal, you should use some zero-copy mechanism such as shared
memory or (vm)splice.
You might be running into some basic scheduler
weirdness here though
and not something inherently wrong with the POSIX queues.
The difference between pipes and message queues is that the latter are
typically used for synchronization, so it's possible that the kernel
tries to optimize for this by doing some scheduling for the receiving
process.
The results with 1M messages had wild variance with
SCHED_FIFO,
SCHED_FIFO is designed for latency, not for throughput. It's no
surprise that it doesn't work well when you have two threads that both
want to grab 100 % of the CPU.
> The first result really has me thinking how much
Jack would benefit from
> using message queues instead of pipes and sockets.
My guess: not at all, because Jack's payload isn't big enough to matter.
Regards,
Clemens
3 p.m.
From: clemens(a)ladisch.de
To: nickycopeland(a)hotmail.com
CC: d(a)drobilla.net; linux-audio-dev(a)lists.linuxaudio.org
Subject: Re: [LAD] Pipes vs. Message Queues
Nick Copeland wrote:
Not sure about that. The CPU(95%) was all in the kernel, not in the process
itself so any improvements to what it scheduled for the process would only
translate into a small percentage difference. Isn't it more likely that the pipe
code is using an inefficient kernel lock on the pipe to ensure it is thread safe?
Please don't misunderstand my 'not sure about', I am relieved to say I am not
a kernel programmer but understanding these kinds of limitations is interesting
as it bears directly on application implementation (see below).
> I got curious, so I bashed out a quick
program to benchmark pipes vs
> POSIX message queues. It just pumps a bunch of messages through the
> pipe/queue in a tight loop.
This benchmark measures data transfer bandwidth. If increasing that
were your goal, you should use some zero-copy mechanism such as shared
memory or (vm)splice.
You might be running into some basic scheduler
weirdness here though
and not something inherently wrong with the POSIX queues.
The difference between pipes and message queues is that the latter are
typically used for synchronization, so it's possible that the kernel
tries to optimize for this by doing some scheduling for the receiving
process. The results
with 1M messages had wild variance with SCHED_FIFO,
SCHED_FIFO is designed for latency, not for throughput. It's no
surprise that it doesn't work well when you have two threads that both
want to grab 100 % of the CPU. It's no surprise that it doesn't work well
It does work very well, just not with piped messages.
when you have two threads that both want to grab 100 %
of the CPU
My system does have 200% available though, it's was dual core and the question
I raised was why there is a scheduling problem between the two separate threads
with pipes whilst it could be demonstrated that there was no real need to have
such contention.
Perhaps I should revisit another project I was working on which was syslog event
correlation: it used multiple threads to be scalable to >1M syslog per second
(big installation). I was testing it with socketpair()s and other stuff. I would be
interested to know if scheduler changes affect it too.
I actually quite like your idea of shared memory - dump a ringbuffer over that and
it could give interesting IPC. Am not going to test that as it would be a significant
change to Dave's code but on the Intel platform it could give some very high
performance without the need for any recursion to the kernel. The event correlator
would not benefit from the use of shmem since it was threaded, not multiprocessed.
Kind regards, nick
3:21 p.m.
From: clemens(a)ladisch.de
To: nickycopeland(a)hotmail.com
CC: d(a)drobilla.net; linux-audio-dev(a)lists.linuxaudio.org
Perhaps I should revisit another project I was working on which was syslog event
correlation: it used multiple threads to be scalable to >1M syslog per second
(big installation). I was testing it with socketpair()s and other stuff. I would be
interested to know if scheduler changes affect it too.
So if the pipe() is replaced with
socketpair(PF_UNIX, SOCK_STREAM, PF_UNSPEC, pipe_fd);
Then the issue I was seeing goes away. Perhaps the pipe() code has not been
optimised since sockets were developed to replace them when IPC suddenly
needed to be between hosts rather than processes? Pure conjecture.
[nicky@fidelispc] /tmp [148] cc ipc.c -lrt -DSET_RT_SCHED=1
[nicky@fidelispc] /tmp [149] ./a.out 4096 10000000
Sending a 4096 byte message 10000000 times.
Pipe send time: 26.131743
Pipe recv time: 26.132117
Queue send time: 18.576559
Queue recv time: 18.576592
The results were repeatable, CPU load was evenly distributed and the ludicrous
context switching figures were gone. Perhaps I should have replaced 'Pipe send
time'
with 'Socket send time'? The message queues seem to maintain the best results.
Somebody should compare that to a shared memory lockless ringbuffer although
I have a feeling they will not exceed the messages queues used here.
Regards, nick.
26 Nov
26 Nov
1 a.m.
On Fri, 2011-11-25 at 15:21 +0100, Nick Copeland wrote:
[...]
So if the pipe() is replaced with
socketpair(PF_UNIX, SOCK_STREAM, PF_UNSPEC, pipe_fd);
Then the issue I was seeing goes away. Perhaps the pipe() code has not been
optimised since sockets were developed to replace them when IPC suddenly
needed to be between hosts rather than processes? Pure conjecture.
Added sockets:
Normal scheduling:
$ ./ipc 4096 1000000
Sending a 4096 byte message 1000000 times.
Pipe recv time: 7.048740
Pipe send time: 7.048648
Socket send time: 2.365210
Socket recv time: 2.365292
Queue recv time: 2.072530
Queue send time: 2.072494
SCHED_FIFO:
$ ./ipc 4096 1000000
Sending a 4096 byte message 1000000 times.
Pipe send time: 5.279210
Pipe recv time: 5.279508
Socket send time: 2.709628
Socket recv time: 2.709645
Queue send time: 5.228892
Queue recv time: 5.228980
Interesting. I find sockets being significantly faster than the much
simpler pipes quite counter-intuitive.
Code at the same location:
http://drobilla.net/files/ipc.c
What would be a more interesting/relevant benchmark is to spawn p
processes, and have each send a message to the next in a chain. Measure
the time from the start until the very last process has received the
message. That should give us a pretty good idea about the sort of
scheduling behaviour we actually care about, and you can crank up p to
make the difference very apparent.
Science is fun.
-dr
25 Nov
25 Nov
9:51 p.m.
On Fri, 2011-11-25 at 13:33 +0100, Clemens Ladisch wrote:
As you say, they are typically used for synchronization, which is
precisely what Jack needs. It seems reasonable that something
specifically designed for realtime synchronous communication might be
better. The payload is nothing.
Of course, for just sync, semaphores would probably be best anyway (they
certainly are in a single process but I have never tested shared ones),
but Jack does not use them because it needs to wait on several things,
which you can't do with semaphores. Message queues are intriguing
because it seems like they could replace both the synchronization pipe
and the server socket, so clients would be using just one wait mechanism
+ communication channel, and a faster one at that.
-dr
Nick Copeland wrote:
> > I got curious, so I bashed out a quick program to benchmark pipes vs
> > POSIX message queues. It just pumps a bunch of messages through the
> > pipe/queue in a tight loop.
[...]
The difference between pipes and message queues is
that the latter are
typically used for synchronization, so it's possible that the kernel
tries to optimize for this by doing some scheduling for the receiving
process.
[...]
> The first
result really has me thinking how much Jack would benefit from
> using message queues instead of pipes and sockets.
My guess: not at all, because Jack's payload isn't big enough to matter.
4528
days inactive
4529
days old
linux-audio-dev@lists.linuxaudio.org
6 comments
3 participants
participants (3)
-
Clemens Ladisch -
David Robillard -
Nick Copeland