[LAU] composing for video

[Danni Coy]

This seems to be due to the massive amounts of IO needed for realtime video

When I was working with YUV4Linux (basically a DVD stream without encryption
or compression) for a 2 GB file I was getting 2 minutes and 18 seconds at
PAL DV resolution. Raw rgb should be almost double that and rgba more than
double. HD resolutions will double and quadruple the amount of
memory/storage needed.

To get data between applications I was using Fifos. Which work, but are
difficult to set up and tempremental at the best of times.

I don't think it would be that realistic to work with compression between
applications in jack type situation. though colour space compression should
be possible

to give some idea of the memory/disk IO requirements needed.
I did some quick calculations

comparing a pal dv sized stream 720 x 576 @ 25 frames a second
vs 48 000 samples per second floating point audio stream.

rgba 32bit float = 864 48k audio streams.
rgba 8bit int = 216 48k audio streams.
rgb 8 bit int =  162  48k audio streams
YCrCb 422 = 108 48k audio streams
YCrCb 420 = 81 48k audio streams

for 720p
rgb 8bit = 360 48k audio streams
YCrCb 420 = 180  " " "

for 1080p
rgb 8bit = 810 48k audio streams

Are there any other color spaces I should be worried about?

Most compositors use RGBA internally either as 8bit int or floats.
FreiOr is a set of video plugins used by a lot of different video editing
programs it prefers rgba 8888, (bgra8888 is the other option) so that format
must at least be acceptable to a lot of video editors. Effectv is the other
shared video plugin set I am aware of and it uses rgb888.
Cinelerra supports RGBA float, RGBA 8888 & YUVA 8888 and non Alpha varients
of the above.

Perhaps the most sensible Idea would be to concider purhaps RGB(A) 888(8)
and YUV(A) 888(8) as different types of port.


On Wed, Dec 2, 2009 at 11:33 AM, Paul Davis <paul at linuxaudiosystems.com>wrote:

> On Tue, Dec 1, 2009 at 8:02 PM, Danni Coy <danni.coy at gmail.com> wrote:
> > You would be suprised. RGBA32 is used quite commonly in the compositor
> > world, usually stored as a sequence of images, with even higher
> precisions
> > available. Raw RGB24 is reasonably common with the video professionals I
> > know, when I questioned them about it they said that they could acheive
> more
> > film like colour grading than they could in any of the YUV colour spaces.
> >
> > It's probably not the best format for realtime work given the amount of
> data
> > that needs to be transmitted (or maybe it is), I have previously worked
> with
> > piping raw YUV (yuv4linux) between applications and having something like
> > jack to manage the connections would be a godsend.
>
> my understanding is that one of the many problems that video faces is
> that there truly is no equivalent to "32 bit floating point for
> audio". whereas this format for audio samples is pretty much
> acceptable for just about all purposes, and the ones not satisfied by
> it are corner cases, in video there are many common cases that prefer
> quite different data representations. for an audio analogy, imagine a
> JACK world where some clients wanted FFT-bin data while others wanted
> floating point encoding of PCM.
>
> until or unless someone can step up and authoritatively say "the
> interchange format for video is XXXXXX", its hard to imagine a "jack
> for video" system really working very usefully for a significant
> number of people. those who work entirely in the RGBA32 or YUV spaces
> would be happy with just that format; anyone mixing processing that is
> better suited to different formats is going to take a hell of a
> performance hit.
>
> --p
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.linuxaudio.org/pipermail/linux-audio-user/attachments/20091202/9b2ae8ee/attachment.htm