[LAD] "enhanced event port" LV2 extension proposal

[Krzysztof Foltman]

Dave Robillard wrote:
> We are definitely going to need some kind of shared data structure
> extension, so really massive message are probably stupid.  16/16 works
> and is un-weird.
>   
So, 16:16 it is.

> 64 bit will become the norm, sure, but does 64-bit alignment matter here
> anyway?  (I have no idea).
I've mentioned pointers. You don't see the use for storing pointers in 
payload?

>   We're probably not going to be firing around
> arrays of floats to do vectorizable things with here
>   
For SSE we'd need 128-bit alignment (4 floats). But I doubt our payload 
is going to be used for SSE and the likes. I'd be very surprised, actually.

> Life is easier and less problematic if we just pick absolute sizes and
> stick with them (so the buffer has a precisely defined format anywhere).
> I doubt this is a compelling enough reason to do otherwise.
>   
Well, usually APIs use pre-determined order, but not pre-determined 
sizes. ie. some fields are 32-bit on some platforms and 64-bit on other 
platforms, but the field ordering is always the same.

> Not really, just casting a char* pointer to a struct to make life easy.
>   
Yes, but... I mean... we'll end up casting to a struct pointer, not 
cherry-pick individual fields via pointer operations like this:

int size = *(int *)(buf + 4);

That's what I was talking about. Accessing a _single_ struct as a 
struct. The "running" pointer will probably be char *, right, but for 
processing of a particular event, it will be cast to a struct pointer.

What I was trying to say was "padding and field sizes - at least for 
some fields - should be platform-dependent". Not in header (because 
header works equally well on 32-bit and 64-bit). I don't want to vary 
field ordering on a per-platform basis either. That'd be a PITA. Just 
field sizes and alignment, and only when *really* needed, in payload. I 
hope you're OK with that. It's what other APIs do, for example, they may 
have a member of type 'size_t' in some structs, which is 32-bit on some 
platforms and 64-bit on others.

Anyway - I suppose at this stage it's a human language-related problem, 
not major differences in opinion.

> We are very precisely defining the layout of a byte array, period.
So, solution number 1. Just look at solution number 2, it might be more 
attractive anyway, even if I wasn't able to convince you by myself. You 
probably need to spend some time trying out your own ideas with both 
solutions. Perhaps I should have given some examples too, I don't know.
> What we have right now is this:
>
> /* 0       4       8       12      16
>  * |               |               |
>  * |       |       |       |       |
>  * | | | | | | | | | | | | | | | | |
>  * |FRAMES |SUBFRMS|TYP|LEN|DATA..
>  */
>
> On the plus side, 4 byte MIDI messages fit nicely in that 16 byte box,
> entire message is 2 64-bit words, 64-bit aligned, no wasted space.
>   
Yes, I got that. It's good and I want to keep it like that. In this 
situation (MIDI data or other data that look the same on all platforms), 
whether you use solution 1 or solution 2, you're fine.

The problems start when you need to put a pointer in an event (not a 
MIDI event; say, a "waveform pointer" event). Which will happen sooner 
than later, I hope.

On 32-bit platform, it will be fine (again, solution 1 and 2 give the 
same result here):

/* 0       4       8       12      16
 * |               |               |
 * |       |       |       |       |
 * | | | | | | | | | | | | | | | | |
 * |FRAMES |SUBFRMS|TYP|LEN|POINTER
 */

On 64 bit platform, though, we would have either:

a) misalignment (bad)

/* 0       4       8       12      16
 * |               |               |
 * |       |       |       |       |       |
 * | | | | | | | | | | | | | | | | | | | | |
 * |FRAMES |SUBFRMS|TYP|LEN| MISALIGNED PTR
 */


It corresponds to solution 1 ("in event type 
http://foltman.com/textpointer, the pointer is placed starting from byte 
0 of payload"). Field placement specified in terms of bytes, no matter 
which platform we're on.

b) padding (good-ish)

/* 0       4       8       12      16      20
 * |               |               |
 * |       |       |       |       |       |       |
 * | | | | | | | | | | | | | | | | | | | | | | | | |
 * |FRAMES |SUBFRMS|TYP|LEN|PADDING|POINTER........
 */

It corresponds to solution 2 ("in event type 
http://foltman.com/textpointer, the pointer is a 64-bit field that 
follows the len field, whatever its natural placement for given platform 
may be"). Ie. this:

struct LV2_WAVEFORM_EVENT {
  LV2_EVENT_HEADER hdr;
  // on 64-bit platforms, the compiler automatically puts 4 bytes of 
padding here
  void *some_ptr; // 4 bytes on x86, 8 bytes on x86-64
};

So, we have 0 bytes padding and offset 12 on x86 (etc) and 4 bytes 
padding and offset 16 on x86-64 (etc.).

Hope the difference between both solutions is clearly visible now.

If I understood correctly, you'd like to start from binary layout and 
*only then* make a C struct based on that design. Right?

I would do exactly the opposite way - design a C (or Pascal, or 
whatever) struct in a way that leads to acceptable in-memory layout at 
least on a typical 32-bit and 64-bit platform (which is easy!).

No, we don't need to do it for event *header* anymore (because the 
layout we pretty much agreed on is the same on 32-bit and 64-bit 
platform). We need to do it when creating new event types (ie. layout 
for payload). That's the part that might not be clear in what I wrote 
previously. I'm trying to imagine both header and typical situations in 
payload too. And from what I see, you seem to agree about the *payload* 
layouts being platform-dependent in some cases (which leads me to 
thinking that defining in terms of bytes is not the way to go for those 
cases).

> The data itself isn't 64-bit aligned, but I'd have to see some real
> reasons to care.
>   
Try reading a 8-byte pointer (64-bit platform) from offset 12. It will 
be equally slow or even crash-provoking as if you read it from offset 11 
on 32-bit platform.

Which doesn't mean that we should use 8-byte alignment for every 
platform. We might use sizeof(void *) alignment, whatever it is locally. 
Any contraindications?

(there is a contraindication, it seems; I've just written a short 
program to check reading/writing a buffer full of doubles from aligned 
and misaligned addresses, and ran it on my machine with crappy Pentium 4 
CPU - seems that reading a double from address misaligned by 4 bytes is 
about 2.6 times as slow than reading from an aligned address, doesn't 
surprise me at all)

Krzysztof