I’ve also tested this under Python 2.6 but it would be nice to see if it also works under Python versions < 2.6 since 2.6 has a couple of compatibility layers built in.

Some of the highlights of this release are:

• Rework API to conform to the new Sender/Reciever API and addressing format
• Supports multi-channel/multi-frame parts of the spec
• Uses js.io class infrastructure
• Generated code now easier to read with decoders implemented in a class hierarchy
• Rewritten dispatch layer – event code now works on the message layer instead of the frame layer even for low level dispatching
• Much closer to implementing 100% of the 0.10 spec
• Infrastructure in place to start working on the 1.0 spec

Give me the code

• tar.bz2 format (sha1)
• zip format (sha1)
• Project homepage & trac: https://fedorahosted.org/kamaloka-js/
• Mailing list: https://fedorahosted.org/mailman/listinfo/kamaloka-js-devel
• git: git clone http://git.fedorahosted.org/git/kamaloka-js.git

About Kamaloka-js

Kamaloka-js is an implementation of the AMQP messaging protocol in native JavaScript. It is setup to be used with Orbited but can be used with any library which produce TCPSockets in the browser similar to Orbited. Kamaloka bindings are generated from qpid XML protocol description files.

Frames

Frames are the basic building blocks of the AMQP data stream.  They contain complete headers that describe queue, track and segment that is currently being constructed.  The payload of a frame (the segment being built) can be broken up into arbitrary sized byte arrays which are then reassembled based on the channel and track they are sent on.  In this way applications with memory constraints can request that frames be no bigger than what the application can fit in memory.  A typical frame header looks something like:

{
  channel: 0,
  track:1,
  is_first_frame: 1,
  is_last_frame: 1,
  is_first_segment: 1,
  is_last_segment: 1
}

Segments

Segments are like frames but instead of an arbitrary split on a data size, each segment is split on struct boundaries. That is to say, when you receive a complete segment you can be sure that it can be fully parsed. There are currently four segment types: A control segment, command segment, header segment and body segment. Command messages are currently the only type that can contain a header and body segment. For instance, the transfer command is used to send messages to and from a queue. It would contain header segments which could be used to route the entire message and it would contain a body segment which contains arbitrary data the user application cared about. Each segment is broken up into at least one frame. Multiple segments would never be sent in a single frame.

Channel and Tracks

A channel is just an integer that denotes related frames and segments. One can think of each channel being a list of incoming frames which are ordered correctly. Once the last frame in a channel is seen, the message is constructed and the channel is flushed and ready to receive a new message. In this way, multiple messages can be received at the same time by utilizing different channels but only one message can be sent on a single channel at a time.

Tracks are an exception to the one message per channel rule. There are two tracks in the current spec. The control track (track 0) and the command track (track 1). Controls preempt commands on a channel, so you can be in the middle of receiving frames on the command track and a control can come in on the control track and you must respond to that first.

This all sounds complicated but you can just think of the channel/track combination as being one entry in an hash. For instance frames coming into channel 0 and track 1 would be given the hash “0.1″:

message_channels["0.1"].add_frame(incoming)

First pass – Frame dispatching

At first it was easier to think of the frame and segment issues as different layers. At the lowest layer I would decode and dispatch each frame and then pass it off to the segment layer once a complete segment had been decoded. The segment layer would then collect the segments, relate them to each other, and then construct the full message. The frame layer looked something like this:

Flowchart showing the frame decoding layer of the kamaloka-js AMQP bindings

The dispatch would then pass it off to the segment decoder.

This became overly complicated because each segment had varying degrees of metadata and the body and header segments didn’t map to the message object very well. I could have gone ahead and created a segment object but I wanted to simplify the code.

Flattening the frame and segment layers

As it turned out flattening the model only added a couple of more steps to the current frame layer. Since frames and segments are just two different ways of breaking up a message for transfer over the wire, combining the two in the same layer made sense. What I ended up with was this:

Flowchart showing how the Kamaloka-js AMQP bindings decode frame and segments into a message

If you notice I now only create a new message if it is the first frame and first segment on a channel. When I see it is the last frame I incrementally decode the segment but I only dispatch the message once the last segment is seen. In the end, these minor adjustments allowed me to strip out a whole layer of redundant code. It also simplified the low level event code as I used to have to manage callbacks for each segment in order to construct a message. Now events only trigger once a full message is received and not when each frame or segment is received.

In kamaloka-js, the JavaScript AMQP bindings, I have been working on implementing this API along with cleaning up the codebase significantly. Today I put the final touches on multi-part frame decoding as well as the dispatching code and hope to have a brand new release next week. Here is a look at both the old and new API:

Old kamaloka-js API as used in release 0.1.0

<script src="/static/Orbited.js"></script>
<script src="/javascript/amqp.protocol.js"></script>
<script src="/javascript/amqp.protocol_0_10.js"></script>
<script src="/javascript/qpid_amqp.js"></script>

<script>
    amqp_conn = new amqp.Connection({host: 'localhost',
                                     port: 9000});
    amqp_conn.start();

    // You should have your server generate a UUID since browser methods
    // are unreliable at best
    session = amqp_conn.create_session('not_a_great_id' +
                                       (new Date().getTime() +
                                        Math.random()));
    var remote_queue =  settings.remote_queue + session.name;
    session.Queue('declare', {queue: remote_queue});

    var queue = session.create_local_queue({name: '0'});
    var output_cb = function(msg) {
        console.log(msg.header.delivery_properties.routing_key +
                    ' sent ' + msg.body);
    }

    queue.subscribe({exchange: 'amq.topic',
                                remote_queue: remote_queue,
                                binding_key: 'com.j5live.#',
                                callback: output_cb});
    queue.start();

    // test a multi segment transfer
    setTimeout(function(){
        session.Message('transfer',{accept_mode: 1,
                                    acquire_mode: 1,
                                    destination: 'amq.topic',
                                    _body: 'Test Transfer From Browser',
                                    _header: {delivery_properties:{
                                                 routing_key:'com.j5live.test'
                                              }
                                             }
                                    });
         }, 10000);
</script>

New kamaloka-js API as used in the upcoming 0.9.1 release

<script src="/static/Orbited.js"></script>
<script src="/javascript/jsio/jsio.js"></script>
<script>
   jsio("import qpid_amqp as amqp");
   jsio("from amqp.protocol import register");
   // load the 0.10 version of the protocol
   register("amqp.protocol_0_10");

    amqp_conn = new amqp.Connection({host: 'localhost',
                                     port: 9000,
                                     socket_cls: Orbited.TCPSocket
                                    });
    amqp_conn.start();

    // You should have your server generate a UUID since browser methods
    // are unreliable at best
    session = amqp_conn.session('not_a_great_id' + (new Date().getTime() + Math.random()));

    var output_cb = function() {
        msg = this.fetch();
        console.log(msg.get('_header').delivery_properties.routing_key +
                    ' sent ' + msg.get('_body'));
    }

    var receiver = session.receiver('amq.topic/com.j5live.*');
    receiver.onReady = output_cb;
    receiver.capacity(0xFFFFFFFF);

     // test a multi segment transfer
     var sender = session.sender('amq.topic/com.j5live.test');
     sender.send('Test Transfer From Browser');
</script>

The new API will offer a lot more control, better dispatching and a simplified interface. Further integration with js.io is planned as well as full support for transactions and flow control.

As I implement the JavaScript API there are some decisions I must make regarding how closely I stick to this new API as implemented in other bindings. Lets look at how the new API receives messages in Python:

...

conn = Connection.open(url.host, url.port)
ssn = conn.session()
rcv = ssn.receiver('amq.topic/org.j5live.demo')

while True:
    try:
        msg = rcv.fetch(timeout=timeout)
        process_message(msg)
        ssn.acknowledge()
    except Empty:
        break
    except ReceiveError, e:
        print e
        break

If you notice the API is very much mainloop based. With every loop we fetch a message off the local queue. If there are no messages, fetch works very similar to the way poll does for sockets – sleep until woken up by network traffic or a timeout. The issue is if we did this in JavaScript we would block the UI and event propagation (newer browsers do expose threading but that feature can not be relied upon in all browsers). If you have ever mistakenly created an infinite loop in JavaScript you will understand how bad this is.

There are two options I see here, keep the fetch method and require the programmer to use timers to read from the local queue or, as currently implemented in my own high level API, attach callbacks to receivers.

The timer/fetch method has a couple of advantages. First it more closely resembles the API of the other bindings. Also it allows tighter management of the local queue. Since each time the timer is triggered we can determine how many messages we wish to process we can give some performance guarantees based on how long it takes to run operations in response to the messages. In this way we can dictate when a message is processed and tweak applications to run smoother. This comes at the price of added complexity which could be even more of a detriment to performance when in the wrong hands.

The callback API has the advantage of being more in line to what JavaScript developers expect. JavaScript is highly event/callback based with onFoo handlers everywhere. We currently work this way. When a message is decoded from the socket we dispatch it immediately to any callback registered to listen for it. The issue here is an application can get hammered with messages with no way to defer processing (except by implementing their own processing queue).

Why not implement both? This is an option but that doesn’t mean it is a good option. Multiple ways of doing the same thing often confuse new users. Understanding the differences between the API’s and the nuances between usecases are often more complex than either API alone. This could be daunting to a new user.

The trick is to make the API’s build on one another. For instance I could add an onReady handler which then allows the user to use the fetch API to grab any messages inside the handler. We would then set up an internal timer if the queue was not completely drained. This would require users be familiar with the fetch API without having to set up their own timer. If they wanted to have more control, they could set up their own timers instead of using the handler.

Any JavaScript developers have any insight as to the best way forward?

The issue is we have diverse communities that rarely agree 100% even among the best of friends. It becomes counter productive when these disagreements become the source of misinformed news articles without general consensus allowed to form within the community itself.

Forget the community for a minute. As individuals we often need to think before we speak in order to make sure we say exactly what we meant to say. The same goes for diverse groups. They must work out the directions they go in in an atmosphere that is free from the chilling effect of having to watch what they say because their intentions may be misinterpreted.

There is also the point of being able to freely express an opinion without fear of outward reprisals. There is a reason Democracy subsists on the concept of private ballots. It allows even the most unpopular opinions to at least be registered while individuals can be sure they won’t be ostracized. The openness comes from being free to dispute results, protest and influence them through public action, not from knowing the individual’s specific vote.

We strive to be inclusive with everyone who has a stake in what we do but when it becomes counter productive one needs to redress the situation so that we continue to move forward. It does no one any good to dwell on speculation from quarters which are ill informed and just looking for traffic grabbing sensationalism. It is perfectly fine to exclude them from discussions they have no stake in and let them feed on the results, judging the community based on its final actions and consensus.

So why do we care about push messaging?

One can think of messaging as a conversation between two people. Poll messaging goes something like this:

Push on the other hand is a bit less chatty:

By eliminating the need to poll to know when an event has happened within Fedora’s Infrastucture, and making it easy for services to push out events in an easy fire and forget manner, we open up the door to a number of interesting possibilities. For instance, instead of implementing mail notification functionality in every service we simply create a mail notification service that listens for events and sends e-mails to people who what them. Do you want to get a notification on your desktop when your build is done instead? This makes it possible to provide that functionality without bogging down the Koji build service.

Human consumable notifications isn’t the only advantage of adding push messaging to our infrastructure. Because the data is first formatted for services to consume, notifications can be used for things like automation and synchronization. For instance someone could write a script that listens for new git checkins at fedorahosted.org and tries to package it into a private repo for personal testing.

What do we need to discuss at FUDCon?

Though this change is minimally invasive and you can ignore it if you don’t need to work with notifications, it is never the less a large undertaking. Some of the things we need to discuss are:

• Usecases – where can we benefit by adding notifications?  How do we envision the notifications will be consumed?
• Payload format – what are the pros and cons of the different ways we can encode and decode data
• Standardization – even though we have a routing protocol we still need to standardize on the type of data to expect
• Libraries – we need to make it dirt easy for infrastructure developers to add notifications to their service
• Performance and security concerns – AMQP is pretty complex so we will need to make sure all of our bases are covered when deploying the QPID servers

AMQP Sessions at FUDCon

There are a couple of talks planned and a hackfest.  Come to the talks to get a deeper understanding of AMQP and how we envision using it within Fedora and then attend the hackfest to help us map out the future of the Fedora Messaging Infrastructure.

Saturday:

• AMQP Messaging for Fedora Developers – come to my session to find out the basics of AMQP messaging and how it is relevant to Fedora’s infrastructure
• AMQP/Qpid on Fedora – The definitive guide – get a more in-depth view of AMQP in Rajith Attapattu session.  He will show you how to setup and configure the Qpid server on Fedora, use the client APIs, and where to go to find help when using AMQP.

Sunday and/or Monday:

• Get on the (Message) BUS Hackfest! – come to Jesse Keating’s hackfest to work out details and hack on the messaging infrastructure.

One of the great parts of Open Source Software is you get to see it develop and grow. It is also one of the biggest misunderstood aspects of such software. In a world where people are gripped by the next best thing – a collective psychosis of product ADD – where patience is no longer a virtue but an outdated notion of an age long gone, evolution is about as exciting as watching paint dry. Just a reminder that even products that seem to just appear overnight, in reality had long periods of closed development to receive polish (and even then they aren’t always great but for some reason people tend to forgive shortcomings in something they bought as opposed to something they got for free).

Knowing the drive behind the developers working on PiTiVi I am confident that in time PiTiVi will become one of the prime examples of FOSS development. For now it is useful enough for some my basic editing needs and every time I try a new version it just gets that much more useful. Keep up the hard work!!!