My only gripe is that yet again my talk’s timeslot seems to have experienced technical difficulty and is not available (the file is only 27min long).  Hopefully that is just a temporary glitch.

It is still an amazing feat of engineering that such as new technology such as WebM codec was used not only to provide archives of the conference but to also stream live.  Not to mention the software and codec used are both Open Source.

Unless you have been living under an Internet rock you already know that Google just released VP8 as an open source and open standard protocol. Almost instantly over night the video codec ecosystem has been turned on its head as Mozilla, Opera, YouTube and Flash, among others have already pledged support and even got tech previews up and running.

It is also no surprise that the MPEG-LA and x264 developers are hitting back. What is of concern is people take these bias sources as fact. The issue is the arguments used to scare us away from moving to a format that they do not control or develop, either hold very little water or apply just as much to using MPEG-LA technologies as it does to VP8 technologies, with the caveat that you have to pay for the privilege.

For instance the assertion that you can never be 100% sure something is patent free (or more precise you have the rights to use a technology without fear of litigation) is on the surface true. Our patent system is fatally flawed in that it is over broad and largely opaque. One can never be sure that some patent might surface or a jury might interpret a patent in a way that is more favourable to the patent holder. The issue with the argument is it applies just as much to H.264 as it does to both Theora and VP8. It is a straw man argument. The logic follows very closely to extortion (legalized because of the flawed patent system) – if you pay us money and use our services we will ignore you but if you go somewhere else we might start harassing you, so you are safe if you stay with us, if you catch my drift. What we do know for a fact is organizations such as Google, Mozilla, Opera and Adobe, all of who have very capable team of lawyers, have come to the conclusion that it is safe to ship the utilize the VP8 codec.

As for the argument that hardware manufactures already support H.264, we wouldn’t get anywhere if we just accepted the status quo. If you look at the list of organizations supporting WebM you will see that they are not only heavy hitters, there is also a good number of them that are hardware vendors. I’m pretty sure we will see android devices coming out in the near future that support the format in the base system (and I’m sure we will see apps and plugins pop up almost instantly).

What about technical arguments that VP8 just isn’t good enough yet? Those arguments came from an x264 developer. (side note – x264 is an Open Source implementation of the H.264 format. While the code itself is under an Open Source license the usage of such code is not open as it is governed by the MPEG-LA patent portfolio). Good enough is a pretty subjective thing. Google seems to think it is good enough for YouTube and as we have seen with the improvements in Theora which started out as a pretty low quality encoder, a little time and effort is all that is needed. Unlike Theora however, which is reaching the upper level of constraints imposed on it by the format, VP8 is at the low end of its potential. With a modern video codec design and more interest due to the open sourcing of VP8, we can expect a stead stream of improvements to the encoder for years to come.

In the end this will all come down to supported codecs. Right now H.264 has the lead but with Google’s YouTube on board as well as all the major browsers which have rising market share, that might not be the case forever. MPEG-LA and H.264 developers will continue to sabre rattle in hopes that their FUD will scare off users instead of actually competing. Here is an idea, why not license H.264 in a more market friendly model? You know one, that doesn’t threaten and extort users and developers? I’m not holding my breath for that one.

Some of the highlights of this release are:

• Rework API to conform to the new Sender/Reciever API and addressing format
• Supports multi-channel/multi-frame parts of the spec
• Uses js.io class infrastructure
• Generated code now easier to read with decoders implemented in a class hierarchy
• Rewritten dispatch layer – event code now works on the message layer instead of the frame layer even for low level dispatching
• Much closer to implementing 100% of the 0.10 spec
• Infrastructure in place to start working on the 1.0 spec

Give me the code

• tar.bz2 format (sha1)
• zip format (sha1)
• Project homepage & trac: https://fedorahosted.org/kamaloka-js/
• Mailing list: https://fedorahosted.org/mailman/listinfo/kamaloka-js-devel
• git: git clone http://git.fedorahosted.org/git/kamaloka-js.git

About Kamaloka-js

Kamaloka-js is an implementation of the AMQP messaging protocol in native JavaScript. It is setup to be used with Orbited but can be used with any library which produce TCPSockets in the browser similar to Orbited. Kamaloka bindings are generated from qpid XML protocol description files.

Frames

Frames are the basic building blocks of the AMQP data stream.  They contain complete headers that describe queue, track and segment that is currently being constructed.  The payload of a frame (the segment being built) can be broken up into arbitrary sized byte arrays which are then reassembled based on the channel and track they are sent on.  In this way applications with memory constraints can request that frames be no bigger than what the application can fit in memory.  A typical frame header looks something like:

{
  channel: 0,
  track:1,
  is_first_frame: 1,
  is_last_frame: 1,
  is_first_segment: 1,
  is_last_segment: 1
}

Segments

Segments are like frames but instead of an arbitrary split on a data size, each segment is split on struct boundaries. That is to say, when you receive a complete segment you can be sure that it can be fully parsed. There are currently four segment types: A control segment, command segment, header segment and body segment. Command messages are currently the only type that can contain a header and body segment. For instance, the transfer command is used to send messages to and from a queue. It would contain header segments which could be used to route the entire message and it would contain a body segment which contains arbitrary data the user application cared about. Each segment is broken up into at least one frame. Multiple segments would never be sent in a single frame.

Channel and Tracks

A channel is just an integer that denotes related frames and segments. One can think of each channel being a list of incoming frames which are ordered correctly. Once the last frame in a channel is seen, the message is constructed and the channel is flushed and ready to receive a new message. In this way, multiple messages can be received at the same time by utilizing different channels but only one message can be sent on a single channel at a time.

Tracks are an exception to the one message per channel rule. There are two tracks in the current spec. The control track (track 0) and the command track (track 1). Controls preempt commands on a channel, so you can be in the middle of receiving frames on the command track and a control can come in on the control track and you must respond to that first.

This all sounds complicated but you can just think of the channel/track combination as being one entry in an hash. For instance frames coming into channel 0 and track 1 would be given the hash “0.1″:

message_channels["0.1"].add_frame(incoming)

First pass – Frame dispatching

At first it was easier to think of the frame and segment issues as different layers. At the lowest layer I would decode and dispatch each frame and then pass it off to the segment layer once a complete segment had been decoded. The segment layer would then collect the segments, relate them to each other, and then construct the full message. The frame layer looked something like this:

Flowchart showing the frame decoding layer of the kamaloka-js AMQP bindings

The dispatch would then pass it off to the segment decoder.

This became overly complicated because each segment had varying degrees of metadata and the body and header segments didn’t map to the message object very well. I could have gone ahead and created a segment object but I wanted to simplify the code.

Flattening the frame and segment layers

As it turned out flattening the model only added a couple of more steps to the current frame layer. Since frames and segments are just two different ways of breaking up a message for transfer over the wire, combining the two in the same layer made sense. What I ended up with was this:

Flowchart showing how the Kamaloka-js AMQP bindings decode frame and segments into a message

If you notice I now only create a new message if it is the first frame and first segment on a channel. When I see it is the last frame I incrementally decode the segment but I only dispatch the message once the last segment is seen. In the end, these minor adjustments allowed me to strip out a whole layer of redundant code. It also simplified the low level event code as I used to have to manage callbacks for each segment in order to construct a message. Now events only trigger once a full message is received and not when each frame or segment is received.

Above is a song I was writing for my sister’s wedding. Unfortunately I never finished it and logistics got in the way of me bringing a guitar to Italy (namely I didn’t want to lug a guitar all around Italy). I thought I would record it using my Cannon PowerShot camera and edit it with Jokosher and PiTiVi to see how far our tools have gone.

I have to say it was fairly easy but I still ran into issues. As far as UI Goes PiTiVi is much easier to use than Jokosher in terms of organizing what I wanted to do. I really only needed Jokosher to tweak the sound a bit but it was still a bit of trial and error to get something decent. I used the high and low pass filters to filter out background noise and echo from my room. Jokosher lacks a basic noise filter effect which can be implemented manually in most other editors by taking a sample of the ambient noise and phase shifting it 180 degrees so the ambient noise is canceled out. Unfortunately the solid bar UI which departs from the usual wave graph makes it impossible to do this by hand. In any case the low and high pass filters worked fairly well to get the most annoying twangs and hisses out.

Another issue is not being able to mark cutin and cutout points and have PiTiVi sync the video with these point. I essentially had to do minimal processing in Jokosher so I could easily sync the video to the edited sound in PiTiVi. It would have been nice to trim the audio in Jokosher and when I imported it into PiTiVi, had it sync up to the correct points in the video. Manually doing that in PiTiVi is hard because of the timeline scale and the fact that it is exclusively drag and drop. It would be nice to be able to enter exact values or load markup metadata so I could snap to my cuts. Once the audio is in PiTiVi we are stuck with PiTiVi’s audio tools which will never be as complete as Jokosher’s (assuming Jokosher gets some development love). It would be nice to be able to jump back and forth between the two.

All that being said, once I planned out what I wanted to do it took no time at all to do it for a video as simple as this. It might not be all roses and honey yet but it is much better than it has been. The next step is to start multi-tracking in Jokosher and syncing it to a video. We shall see.

In the next few days I will be concentrating on documenting the design of the bindings so anyone who wants to can easily jump in and contribute patches as well as suggestions. Please visit the project page to find out more and see how you can help out.

What’s in store for the future?

• High level bindings for each of the major AMQP client patterns to get developers up and running with AMQP in the browser fast
• QMF bindings for working with the QMF management console

Where can I find this release?

• kamaloka-js-0.1.0.tar.bz2 (sha1)
• kamaloka-js-0.1.0.zip (sha1)

What’s in this release?

• Initial release
• Connection handshaking works
• AMQP lowlevel bindings for version 0.10 of the AMQP Protocol mostly complete
• Higher level bindings implementing the publisher/subscriber pattern is ready for use

How can I find out more?

• Project page: https://fedorahosted.org/kamaloka-js
• FAQ: https://fedorahosted.org/kamaloka-js/wiki/Faq
• Mailing list: https://fedorahosted.org/mailman/listinfo/kamaloka-js-devel

Kamaloka-js is an implementation of the AMQP messaging protocol (http://amqp.org/) in native JavaScript released under the MIT license. It is setup to be used with Orbited (http://orbited.org/) but can be used with any library which produce TCPSockets in the browser similar to Orbited. Kamaloka bindings are generated from qpid XML protocol description files.

One thing to note is Chromium’s integrated debugger is a lot faster than Firebug. I haven’t used it extensively but if it can break at breakpoints and introspect objects without getting screwy I might end up switching. I really like firebug but lately it has become dog slow. It usually takes a couple of minutes of waiting for each frame of data to be processed and displayed while under Chromium with their debugger on, the slowdown is noticeable but not significant compared to when the debugger is off.

UPDATE: Chromium’s debugger is probably faster because it doesn’t actually do anything that I can see. It displays scripts but doesn’t actually break into the code

This along with QMF will give us the ability to send out notifications of events, monitor statistics and eventually supplement REST and XML-RPC interfaces with more efficient multi-response method calling (imagine streaming rows of data instead of waiting for the entire row set to be sent).

AMQP Handshake over Orbited

Above is a little hex dump web app I wrote to test the bindings. What you see is the start of the AMQP handshake. First I send the header indicating the version of the protocol I am expecting. I then get the ’start’ response back stating the capabilities of the broker I have connected to. I then send the ’start-ok’ packet indicating what auth mechanisms I support and the locale (I would also send a token if I were using a particular auth other than anonymous).

That is about as far as I have gotten but since the bindings are autogenerated from an XML description of the protocol most of the control and command classes are implemented. There are still some types I am trying to figure out how to support (the supported types are a magnitude above what was needed for D-Bus) and I need to smooth out the rough edges as well as write the higher level user bindings but I am off to a good start.

Code will be posted as soon as the project resources are approved and setup. Keep watching my blog for updates.

BTW: The slow response times in the time stamps are due to running inside of the firebug debugger and writing out the hex dumps on the fly using JQuery. It seems the more sophisticated JQuery and Firebug get, the more they butt heads.

I can say without a doubt the turnout was amazing and even though not everything I heard all weekend was positive it was a giant leap forward in then understanding of the importance of Open Video and culture. I won’t put a figure on how many people attended but some of the upstairs talks were standing room only and after the first day some of the organizers were lamenting that then needed to get bigger rooms (consequently some of the talks were swapped the next day). Speaking about the organizers, they ran an incredibly smooth ship and should be thanked and praised for their efforts.

The Good

Apps

I was mainly there looking to see what video producers wanted from FOSS application developers and to support the PiTiVi/GStreamer teams on behalf of the GNOME Foundation. It is amazing to see the PiTiVi non-linear video editing app at such a usable state. While Edward Hervey (bilboed on irc) gave his mini presentation on PiTiVi I was busy hacking up a “How To Make Chocolate Truffles” video from pictures and clips I had laying around.

Afterwards I showed him some of the bugs I encountered in the 0.13.1 release and he just rattled off, fixed in git, fixed in git, fixed in git…etc. Sadly the releases are tied to GStreamer releases (which is a good thing from a development/bugs standpoint but not so good from a user standpoint given the early stages of PiTiVi) so we won’t see an official release soon. I plan on trying to automate a Fedora Repository at some point just to be able to view the progress without breaking my system.

The point is PiTiVi is about 90% there (and perhaps 100% in git) to be able to support my needs for basic video editing in terms of stability and basic tools. That should be pretty reflective of those who need to do things like screen casting and interview style video blogs. Some advanced features like effects (look at Cheese for some examples of this already working in an app) already exist in GStreamer and just need to be integrated in PiTiVi’s UI and rendering pipeline.

There was also a show of Cinelarra but more interesting is the GTK+ fork Lumiera which unfortunately is not usable yet but the direction they are going in (GTK+ interface and some GStreamer integration) looks like a great re-start in the case of pro level editing tools.

Also of interest in the pro level space was Blender which seems to be the pro project with the most momentum and features for pro’s. At least that was the initial reaction from some on the Red Hat media team. The dev’s did admit that the functionality is limited to what they needed during production of Big Buck Bunny (and other productions currently in the queue) but that in those areas it is rock solid. It is interesting to see a UI designed with different usability profiles. For instance one of Blender’s usability criteria is the avoidance of repetitive strain injury. To combat RSI mouse clicks are evenly divided between left and right mouse clicks.

Bassam Kurdali, one of the Blender developers and animators, came up to me later in the conference and said he had noticed me using PiTiVi to edit my video. He was impressed at the simplicity and slickness of the interface and how far along it is. There is plenty of room for different approaches and a real potential for cross pollination between the pro tools and the every day end user tools.

What Content Producers Want

Speaking of end users we got to hear from a bunch of them who let us know how we could support them. One of the biggest themes was that Windows tools suck and those who taught others couldn’t just tell them to go out and buy a mac (praises were heaped on iMovie and Final Cut Pro). They really want an easy to use tool, with the unfortunate note that it would have to run on Windows. One really good thing is that a lot of the non-tech content producers understood the need for free codecs. However in the end they just want a simple way to render down to DVD, You Tube, Daily Motion, iPhone, etc. and don’t want to deal with formats.

I ended up collecting a bunch of buisness cards and am toying with the idea of starting a feedback group with content producers which would get them involved in improving GNOME App usability from the perspective of those who are not yet familiar with the GNOME workflow. If we are serious about expanding our reach we need to go beyond our current self selecting internal feedback loops. The goal wouldn’t be to get these people using GNOME (though giving them a way through the apps wouldn’t be a bad thing). It would be more about getting groups outside of GNOME/Linux to be part of the process of improving GNOME. Will it be fruitful? I don’t know but it is an interesting experiment with a potential huge payoff for a little bit of effort.

Sita Sings the Blues

This good section wouldn’t be complete without the mention of Sita Sings the Blues by Nina Paley which is a feature length (82 minutes) animated film released under the Creative Commons Attribution-Share Alike 3.0 United States License. You have complete rights to watch, screen, remix and redistribute the film as long as you abide by the license. I do suggest you watch it and if you like it buy the DVD or simply donate to encourage more works like this (I bought the DVD for $20). Not only is Nina a content producer but she is heavily involved in advocating her distribution methods, going as far as documenting the process that went into releasing Sita under a creative commons license and in her work with QuestionCopyright.org.

Mozilla and the Open Video Contest

I was very impressed with Mozilla’s involvement and their push for Ogg Theora to become a base line codec for the HTML 5 video tag. They are also helping launch the upcoming Open Video Contest which would see the winner flown to the 2010 South by Southwest conference. We should probably run some sort of sister contest to encourage GNOME users to submit entries.

The Bad

It wasn’t all roses. While I feel we are reaching independent content producers way more than I would have though at this point, some of the big companies still don’t get it or are afraid of Open Video implications.

Adobe

It must be said that Adobe has been somewhat good at working with the community over long periods of time but that they just never get around to resolving key issues. What really surprised me was when on one of the industry round tables the Adobe representative pointed to their release of the Flash documentation as a shining example of this relationship. After checking with a developer of an alternative flash implementation I was told those documents are pretty much useless. Due to bugs, some of the spec just doesn’t work as written and other issues makes it impossible to write a third party Flash player.

YouTube/Google

While reportedly Chrome will ship with Ogg Theora support their flagship video site YouTube seems afraid to do so. Their rep at the round table stated some pretty audacious things such as continuing the myth that Theora wasn’t good enough when clearly that argument was directly debunked (the side by side comparisons were even playing on HDTV’s at the conference).

Even more of an issue was the representative’s idea on what Open Video meant. He declared that they would love to support Open Video but that it meant letting anybody do whatever they wanted and that doesn’t work from a buisness perspective.

Open Video isn’t about wild west, trample on rights. If anything it is about balancing the rights of content producers, end users and fair use. From what I read, YouTube’s position is that they are the 1000 pound gorilla in video distribution and at the end of the day they only believe in a user’s and content producer’s freedoms if it is walled behind their own servers. “All the world’s video” indeed.

The solution there is to drive traffic to sites like Daily Motion and Blip.tv which understand the issues involved.

Conclusion

Nothing is perfect, but we are off to a really good start. In the end it is up to us to keep the momentum going and eventually produce a better experience within the complete Open Video stack, from content production to delivery. The web was built and exploded around the concept of open technology. Let’s continue to make sure this is the case going forward. The last thing we want is the web to become the domain of a few, with creativity being stifled by restrictions in the non-open parts of the stack.

The interesting part of this conference is it isn’t exclusively a tech conference. According to their website:

Open Video is a broad-based movement of video creators, technologists, academics, filmmakers, entrepreneurs, activists, remixers, and many others. When most folks think of “open,” they think of open source and open codecs. They’re right—but there’s much more to Open Video. Open Video is the growing movement for transparency, interoperability, and further decentralization in online video. These qualities provide more fertile ground for independent producers, bottom-up innovation, and greater protection for free speech online.

If we have any hopes of success in bringing the general public an understanding of our views on the importance of openness and freedom, forming a larger community with like minded content creators is the next logical step and another piece of the puzzle.

I will be attending the conference as a representative for GNOME. Others should consider registering and attending to represent their own media minded project. I hope to see you there.