[LAD] Tracing malloc() and mutex usage in realt-time software

Heya! Not sure if anyone of you already read this announcement about my mutex profiler "mutrace" I wrote a couple of days back: http://0pointer.de/blog/projects/mutrace.html Since I wrote that blog story I have added a couple of features to mutrace that are particularly relevant when developing real-time applications. I'd like to draw your attention to these, since this might be of some interest for the audio community. "mutrace -r" can be used to track down any mutex usage in real-time threads. As everyone probably knows it's a good idea to abstain from mutex usage entirely in RT threads. However, actually doing that is not always realistic, and sometimes the more complex your project gets the harder it becomes actually making sure that no hidden mutex usage creeps into your program. Too verify lock-freeness or track down the culprits you may use "mutrace -r". It will tell you not only if am RT thread used a mutex, it will also tell you how often it was contended and how often it changed ownership (i.e. how big the risk of contention is even if it didn't contend in the specific run.) In addition it will show you the mutex protocol, so if you are using a mutex from an RT thread and really cannot do without it, you can at least make sure it is a PRIO_INHERIT mutex, And there's also now a second tool called "matrace" in the tarball that tracks down malloc() usage from RT threads. It's a very simple tool, which just catches the memory allocations/frees, checks the scheduling and complains if necessary. With these two tools two of the biggest donts in RT programming may be tracked down: use of mutexes and use of malloc. How does the output look like? As an example I simply ran jackd through mutrace -r and matrace, for 10s without doing anything else. The result for mutrace you may find here: http://0pointer.de/public/mutrace-jackd.txt (The interesting table is at the end, so scroll down) Not sure what to make of this, as I don't know the Jack internals too well. However mutrace reveals that at least one mutex was used from an RT thread and even contended. Moreover that mutex was not PRIO_INHERIT. And here's matrace: http://0pointer.de/public/matrace-jackd.txt Seems free() is called a couple of times from an RT thread. These might, or might not be problems. The tools can just inform you what is done. If there's something to fix is up to the eye of the beholder. Get the tool here: http://git.0pointer.de/?p=mutrace.git Have fun, Lennart -- Lennart Poettering Red Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4