The problem with static data flow schedules

This post is just a quick note. I’ll extend on it later.

Imagine you have a data flow graph with nodes A, B, C, D, G. Of these nodes, A and B are a source of stream and G is a sink. Now, say, the static scheduler gives you this iterative list of firings: B, A, C, B, A, G, B, D, D, G, D, A, G, C, B, G

Now, your application is real-time and, as usually, it is call-back based (the hardware awakes a process in a regular basis providing and consuming data).

Data flow scheduling is very interesting so far, but a question arises: In practice how can you use this scheduling from the call-back process?

Actually the obvious solution I can think is not good at all: you can do call-back adaptation of a blocking interface. That implies leaving the call-back process to just do buffering, and have another thread that executes the data-flow. Where source and sink nodes (A, B and G) can be blocking if data is not available. But in most cases minimize buffers on the inputs and ouputs is desired. So you want the scheduler that collects input as regularly as possible. This is an admitted (SDF scheduling paper, section V.C) limitation of the model.

I haven’t found yet any research focusing on that problem (though I admit I still have a bunch of related papers to read). So any feedback on this will be greatly appreciated. However, I have some ideas on how the solution could be, I’ll post about it later.

Synchronous data flow definition

In CLAM related papers we use to define the synchronous flow as a flow where messages get produced and consumed at predictable —if not fixed— rate. This definition relates very well with my intuitive idea of synchronicity. However, having seen examples of multi-rate data flows and having implemented dynamic schedules, I didn’t feel very comfortable with this definition. The main concern is that node’s firing does not necessarily need to be predictable.

Recently, reading some papers from Edward Lee and others I saw the light:

A synchronous data flow is a graph of synchronous nodes. Where a node is synchronous if we can specify a priori the number of input samples consumed on each input and the number of output samples produced on each output each time the node is fired.

This is a extremely simple property of the graph itself and it is independent of its scheduling strategy. But the point is that it is not so intuitively related with the synchronicity idea. This great paper from Lee and Messerschmitt shows (very formally) how to know if an iterative scheduling exists or not. The condition is simple: the rank of the graph’s incidence matrix must be the number of nodes minus one. Also, if such scheduling exist, there is an algorithm that finds it. The algorithm itself is simple to understand but not so to implement since it involves linear equations solving and exponential search.

To make it clear, an scheduling is a list of executions that gets repeated —being for one or many processors—, and necessarily needs to be iterative since an infinite stream is assumed. Of course, an iteration is a list that may include some or all the nodes many times.

Avoid making decisions. Linus dixit

I just discovered this article on Linux kernel management style written by Linus Torvalds. It’s amusing and ironic in the best Linus style. For example, his first recommendation:

First off, I’d suggest buying “Seven Habits of Highly Successful
People”, and NOT read it. Burn it, it’s a great symbolic gesture.

I think these paragraphs summarize quite well the core of the philosophy:

So the name of the game is to avoid decisions, at least the big and painful ones. Making small and non-consequential decisions is fine, and makes you look like you know what you’re doing, so what a kernel manager needs to do is to turn the big and painful ones into small things where nobody really cares.

It helps to realize that the key difference between a big decision and a small one is whether you can fix your decision afterwards. Any decision can be made small by just always making sure that if you were wrong (and you will be wrong), you can always undo the damage later by backtracking. Suddenly, you get to be doubly managerial for making two inconsequential decisions – the wrong one and the right one.

Later he talks about how to put blame and criticize. I do strongly agree on how sense humor plays an important role here.

Some humor can help pad both the bluntness and the moralizing. Going overboard to the point of being ridiculous can drive a point home without making it painful to the recipient, who just thinks you’re being silly. It can thus help get through the personal mental block we all have about criticism.

Automatic releases: get the right version

There is one topic I’d like to do a series of small articles: automatic releases. This is something we have been improving more and more in CLAM and now I feel we are getting it right. In a nutshell, managing a CLAM release used to be very time consuming (it could take months!) now it’s like pressing a button.

We’ve done a lot of things: keep the code stable in all platforms (automatic tests and testfarm is a great help here), automate the process of generating all release files which includes source tarballs, mac-osx and windows installers, debian package, and more. Of course, the apps must have the version embedded (available in the about box, for instance). And finally, all those files have to be published accessible and well organized in the web.

And there’s more complexity to manage here: we provide “subversion” or “development” releases on one hand and “stable” releases on the other. The first ones look like CLAM-NetworkEditor-0.4.3_setup.exe, the second ones look like CLAM-NetworkEditor-0.4.4~svn09719_setup.exe. This last long version means: this is a svn snapshot of NetworkEditor (revision 9719) in the way of the stable version 0.4.4.
And of course, we are managing versions of four applications and one set of libs. Thus, the complete version of one app includes the CLAM lib version.

On the lowest level of this infrastructure we have the scripts that automatically retrieve the correct version. Today I’ve finished a code clean-up of the python module we are using. The main functions of the versionInfo.py module is this:

def versionFromLocalInfo( product='CLAM', changesFile="CHANGES" )


Keep reading to learn how it works.

Dreamhost

I’ve lately become a Dreamhost client. I host my personal web, my band’s web and, soon, the org.lliure.cat web. The last is an organization that campaigns and promotes free-software in the Catalan administration. Well, so far I’m very happy with it and I highly recommend it. They are very cheap and have grate customer support and community. The site is entirely run by free software. Debian is the operating system, you have Apache, of course and many great web publishing systems: WordPress, Mediawiki, Gallery. Other goodies include php5, mailing-lists, etc.

They have an interesting discount schema. Enter my referral code: PAUARUMI when you sign up and you will save 60eur ($77). In return I get 15eur ($20) referral discount. I used this system when I joined and it makes the first year only 33eur ($42). Absurdly cheap! All clients can create referral codes, when you create one you have to decide how to split $97 between the you and the referee.

CLAM roadmap

It was quite a long time from my last blog (and planet) entry, and I feel its a good time to explain how things are going on the CLAM development trenches and what core developers have been discussing about the mid-term roadmap (to 1.0 and post 1.0). We wrote a first version of the roadmap in the wiki.

We are still mostly concentrated on having good mac osx binaries. Intel and ppc dmgs are ready to download and test. So I’d say that we are almost there. But before doing a stable release, some things needs to be fixed, specially for powerpc architecture. So, the more testing and bug reports we can get, the better. If you own a mac, please drop by clam-devel and say how applications are behaving. If you don’t, well, do the same; we are specially interested in having a rock-solid NetworkEditor and we want to do a lot of bug-hunting before 1.0.

And now for the roadmap. The following is a stripped-down version of the one in the wiki, but reflects the main goals.

1.0 CLAM release (very short term)

• Due by the first or second week of February
• Main goals: installers for the three platforms (lin, mac, win); get clam applications and libs included in major linux distributions (ubuntu and debian, at least. Maybe fedora core); provide usage videos and announce the release widely. At least two quick and minor release will happen before 1.0 to get more users feedback.
• Tasks to be done includes: Bug hunting session, edit a already recorded video showing NetworkEditor and Prototyper, Fix some problems with Mac OSX applications, and try to get into some Fedora Core repo (planet-ccrma should be the one) and finish inclusion to debian and ubuntu.

1.x CLAM release series (mid term)

• The 1.x releases roadmap will be refined when we release 1.0
• Due by March
• Three main goals: code clean-up, improve clam extensibility mechanisms, and add (many) useful examples for a wider user base.
• Some task to be done (not complete): have a plugin architecture for clam processings; remove unneeded modules and deprecated code; reorganize directories to better match built libraries, simplify the build system (scons); make the addition of new processings (in a plugin) extremely easy; tutorials on how to extend clam; widgets, audio backends and file-formats should be plugins, not dependencies; develop a broad suite of alsa and vst plugins with user-interface (maybe audio-unit also)

There are lots and lots of ideas and features for CLAM 2.0. A long-term roadmap will be defined during 1.x development. Of course, feedback from the users will be very important.

Guillamino band on tv

Pau Guillamet just discovered this short video from a local tv. We are the guillamino band playing at L’Espai theater in Barcelona. It is the more formal gig we have done so far (I mean: people seated, serious sound system…). Yes, I’m the guy at bass and synth. The PG interview is in catalan and most of the music is from the record, not live.

The amazing thing is that I didn’t notice anyone with a camera on that gig!

(I don’t know how if planet accept embedded youtube html. In this case just follow this link)

Pingbacking about Ubuntu

I have to confess I’m new to blogs and I’m learning some of its very common features. An interesting one is the pingback or traceback protocol. The new thing for me was to learn that a normal link to another blog entry is all you have to do to get linked back. For example, yesterday I read in ubuntu planet a blog entry asking for people blogging about ubuntu (and here am I).

Since I’ve placed a link to that entry in the previous paragraph it will automatically appear as a comment there (that’s what I expect, at least). For that to happen, both blogs have to be pingback enabled. I’ve just learnt that pingback is basically the same as traceback but less spam prone. I guess it requires a hand-shaking between both blog systems before publishing any comment.

I’m also happy to see that wordpress creates an independent feed for every category. For example:
http://parumi.wordpress.com/tag/ubuntu/feed

This comes very handy when you participate in a planet and do not want to export all the posts: you just reserve a tag for that planet. Well, anyway I’ll keep feeding everything to the clam planet. No high-traffic problems by now :-)

Getting firewire audio work on Linux

My requirements for an audio interface where basically: low-latency, good pre-amps and DA/AD converters, and having more than two outputs. And well it had to be an external interface to use with my laptop (macbook), work with Linux, and below 400 eur.

I need low-latency for live gigs. I use to play bass and some synths with Guillamino, but I really want to incorporate soft synths, and live material processing with sooperlooper, supercollider and freewheeling.

For latency reasons I discarded buying a USB interface. Also it’s a plus that nowadays music shops have plenty of new firewire interfaces.

When recording music, monitoring or hearing an audio mix that is independent of what is being recorded is a must. Since I’m aiming to have a really portable recording station I don’t want to rely on external mixing desk. So I needed at least 4 outputs and flexible routing.

Firewire support for Linux is in its early stage but growing steadily. I knew about the freebob project when Pieter Palmers presented it at Linux Audio Conference 2005. Freebob is a Linux driver (by now only a JACK backend; alsa driver is planed) for devices based on the bebob chip. Fortunately many audio interfaces are based on this chip.

My choice: Focusrite Saffire (LE)

Edirol FA-101 and ESI QuataFire 610 also looked good and where options I also considered. But in comparison, Saffire bought more features (and hopefully sound quality) for its price. I’ve to say it is fantastic to have a huge two-story audio-gear supermarket (Alfasoni) near my flat in Barcelona. And their prices are generally lower than any other store in the city. Saffire (LE) was 323 eur plus taxes. Not bad.

Saffire (LE) audio interface have 6 ins and 8 outs (including digital spdif), and good quality audio (at least this is my first impression in a home-studio environment, and Focusrite have a reputation on its high-end gear).

In Linux, you don’t have an app for controlling the internal audio routing, but you can access all 6 ins and 8 outs with JACK, and with a ridiculous latency you can do all monitoring routing with JACK, so no big deal. Another happy surprise was to find that firewire provides enough power for the device. You only need to plug the power adapter for enabling phantom feed for condenser mics.

How to enable freebob driver in Ubuntu

That can be a little tricky. If your distribution doesn’t come with libfreebob package you’ll need to compile this library (and some dependencies) from the source and then rebuild libjack from subversion sources. But that’s not the end of the hassle. You’ll probably find (as I did) that the new compiled (svn) libjack is not compatible with already installed libjack and applications (I think they use different client/server protocols so the incompatibility have sense). Of course, I was not happy at all having to recompile all my jack applications so I looked at fixing the libjack .deb package from its source package.

Luckily, I found that enabling freebob in Ubuntu Feisty is really easy. Feisty is currently the testing/developing Ubuntu version, due for april 2007, so it’s not intended users who need a solid, reliable OS. However, since I have a macbook and the Feisty kernel supports macintel hardware much better than current Ubuntu Edgy, installing Feisty was a decision I already took. And I’ve had no crashes and had a good experience, so far.

So, the good news is that in Feisty you already have libfreebob0 in universe repository. The bad new is that, however, jackd comes with freebob driver disabled. Note that freebob is not a driver you can plug-in but is it built-in with jack. I believe it’s a packaging bug and very simple to resolve (though still in unconfirmed state).
Update: (January 25th) it seems that this problem has been resolved in new ubuntu-feisty libjack package (its already compiled against libfreebob). So ignore the next steps for compiling jackd from sources.

So here goes the how-to for enabling freebob in jackd maintaining all jack-apps compatible:

Enable universe binary and source repositories and install libfreebob-dev
$ sudo apt-get libfreebob0-dev
Download and compile libjack from source package. It can be done with a single instruction:
$ sudo sudo apt-get -b source libjack0.100.0-dev
Is very likely that you’ll need to install several -dev packages for compiling this source package. But don’t worry, the error message specify all those packages. After installing them, try again.
Then install all new created .deb packages:
$ sudo dpkg -i *.deb

Now if you do
$ jackd -d freebob
you’ll see that –though it still doesn’t work– at least it is using the freebob driver.

The rationale behind that is that the autoconfiguration of libjack enables freebob driver if it finds the the freebob library and headers installed. But libfreebob0 dependency is not enforced in its debian/rules, that’s why binary package have freebob disabled. And since Feisty already have libfreebob I think it should be enforced.

Fixing /dev/raw1304 permissions

Let’s fix the before-mentioned error. Doing jackd -d freebob I got:

Freebob using Firewire port 0, node -1
Ieee1394Service::initialize: Could not get 1394 handle: Permission denied
Is ieee1394 and raw1394 driver loaded?
Fatal (devicemanager.cpp)[68] initialize: Could not initialize Ieee1349Service object
Fatal (freebob.cpp)[69] freebob_new_handle: Could not initialize device manager
FreeBoB ERR: FREEBOB: Error creating virtual device

That’s because a normal user does not have access to raw1394
$ ls -l /dev/raw1394
crw-rw---- 1 root disk 171, 0 2006-12-17 16:10 /dev/raw1394

Changing permissions
$ sudo chmod o+rw /dev/raw1394
solves the problem. However after a new reboot the permissions have been reset. A permanent and easy way to solve this is adding your user into the disk group.
$ sudo gedit /etc/group
In my case, the username is parumi so I edited the group file like this: “disk:x:6:parumi”
Update: $ sudo adduser parumi disk is the right way instead of directly editing /etc/group. Thanks Maarten de Boer for this feedback.

Another solution is using udev rules like explained in this linux-audio-user thread

After that, login again and you should be able use jackd -d freebob and all installed jack-apps. Finally, this screenshot shows qjacktl connection window with the Focusrite Saffire (LE) 6 inputs and 8 outputs.

Building Ardour2 in Ubuntu Feisty

Disclaimer: Ubuntu Feisty is still in its first weeks, so might be a very unstable distribution. But in my case is working decently well so far (I’m using Feisty because it comes with a kernel more mac-intel friendly). Anyway, de howto will probably work in Ubuntu Edgy.

Ardour is a wonderful digital audio workstation developed mainly by Paul Davis. The new version Ardour2, which still in beta stage, looks and feels really pro. It’s probably my favorite Linux app.

Now, how can you build the bleeding edge (subversion) version of ardour2 in Ubuntu? (tested in Feisty)

Important: Please refer to the official ardour download and build page for more detailed and updated information. This is just a quick receipt for Feisty.

1. Get the code from subversion
svn co http://subversion.ardour.org/svn/ardour2/branches/2.0-ongoing ardour2.0-ongoing
cd ardour2.0-ongoing

(Updated 2007/05/15 changed trunk for branches/2.0-ongoing . After release 2.0, trunk needs a newer jack version not available in Feisty, and it will be under big changes for a while, so it’s not the recommended version for the normal user)

2. Install needed tools and build dependencies:
sudo apt-get install -y scons build-essential libtool pkg-config gettext libjack-dev libxml2-dev libsamplerate-dev libraptor-dev liblrdf-dev libgnomecanvas2-dev libboost-dev liblo-dev libglib-dev libgtkmm-dev libsndfile-dev libasound2-dev automake

(Updated 2007/05/01 added libasound2-dev and build-essential)

3. compile and install
scons
sudo scons install

Tip: use scons -j3 if your computer is a dual core.

4. start jack with qjackctl and run ardour2

And ready to use ardour2 in all its glory:

In this screenshot you can see that I’m using a 3d desktop (windows shadows). Amb using compiz-freedesktop-gnome which already comes in the Feisty universe.