Work in progress

This is a draft document which tries to give you an overview how new technologies will be integrated in aRts. Namely, it does cover the following:

This is work in progress. However, it should be the base if you want to see new technology in aRts. It should give you a general idea how these problems will be adressed.

However, feel free to correct anything you see here.

Things that will be use aRts technology (so please: coordinate your effords):

how interfaces work

MCOP interfaces are the base of the aRts concept. They are the network transparent equivalent to C++ classes. Whenever possible you should orient your design towards interfaces. Interfaces consist of four parts:

These can be mixed in any way you like. New technologies should be defined in terms of interfaces. Read the sections about asynchronous streams and synchronous streams, as well as the KMedia2 interfaces, which are a good example how such things work

Interfaces are specified in .idl code and run through the mcopidl compiler. You derive the Interfacename_impl class to implement them, and use REGISTER_IMPLEMENTATION(Interfacename_impl) to insert your object implementations into the MCOP object system.

codecs - data-decoding

The kmedia2 interfaces allow you to ignore that wav files, mp3s and whatever consist of data streams. Instead, you only implement methods to play them.

Thus, you can write a wave loading routine in a way that you can play wave files (as PlayObject), but nobody else can use your code.

Asynchronous streams would be the alternative. You define an interface which allows you to pass data blocks in, and get data blocks out. This looks like that in MCOP:

interface Codec {
  in async byte stream indata;
  out async byte stream outdata;
};

Of course codecs could also provide attributes to emit additional data, such as format information.

interface ByteAudioCodec {
  in async byte stream indata;
  out async byte stream outdata;
  readonly attribute samplingRate, bits, channels;
};

This ByteAudioCodec for instance could be connected to a ByteStreamToAudio object, to make real float audio.

Of course, other Codec types could involve directly emitting video data, such as

interface VideoCodec {
  in async byte stream indata;
  out video stream outdata;      /* note: video streams do not exist yet */
};

Most likely, a codec concept should be employed rather than the "you know how to play and I don't" way for instance WavPlayObject currently uses. However, somebody needs to sit down and do some experiments before an API can be finalized.

video

My idea is to provide video as asynchronous streams of some native MCOP data type which contains images. This data type is to be created yet. Doing so, plugins which deal with video images could be connected the same way audio plugins can be connected.

There are a few things that are important not to leave out, namely:

My idea is to leave it possible to reimplement the VideoFrame class so that it can store stuff in a shared memory segment. Doing so, even video streaming between different processes would be possible without too much pain.

However, the standard situation for video is that things are in the same process, from the decoding to the rendering.

I have done a prototypic video streaming implementation, which you can download here . This would need to be integrated into MCOP after some experiments.

A rendering component should be provided that supports XMITSHM (with RGB and YUV), Martin Vogt told me he is working on such a thing.

threading

Currently, MCOP is all single threaded. Maybe for video we will no longer be able to get around threading. Ok. There are a few things that should be treated carefully:

However, what I could imagine is to make selected modules threadsafe, for both, synchronous and asynchronous streaming. That way - with a thread aware flow system, you could schedule the signal flow over two or more processors.

This would also help audio a lot on multiprocessor things.

How it would work:

synchronization

Video and midi (and audio) may require synchonization. Basically, that is timestamping. The idea I have is to attach timestamps to asynchronous streams, by adding one timestamp to each packet. If you send two video frames, simply make it two packets (they are large anyway), so that you can have two different time stamps.

Audio should implicitely have time stamps, as it is synchronous.

dynamic composition

It should be possible to say: An effect FX is composed out of these simpler modules. FX should look like a normal MCOP module (see masquerading), but in fact consist of other modules.

This is required for aRtsbuilder.

gui

All GUI components will be MCOP modules. They should have attributes like size, label, color, ... . A RAD builder (aRtsbuilder) should be able to compose them visually.

The GUI should be saveable by saving the attributes.

midi

The MIDI stuff will be implemented as asynchronous streams. There are two options, one is using normal MCOP structures to define the types and the other is to introduce yet another custom types.

I think normal structures may be enough, that is something like

struct MidiEvent {
  byte b1,b2,b3;
  sequence sysex;
}

Asynchronous streams should support custom stream types


back to index