Well, I’ve gone and done it. Thanks to David Malcolm’s excellent 2to3c tool and some hand wrangling with PyUnicode objects I was able to get D-Bus Python compiling and working on Python 3. Grab the patch and start testing it out.
I’ve also tested this under Python 2.6 but it would be nice to see if it also works under Python versions < 2.6 since 2.6 has a couple of compatibility layers built in.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]I am proud to announce the second release of the kamaloka-js AMQP bindings. This release brings with it a high level API modeled after the new Sender/Receiver and addressing API’s being developed in the Qpid project. It is intended to bridge the gap between the 0.10 protocol and the upcoming 1.0 protocol. I have also simplified the code base and improved the code generator. Because we are getting so close to implementing the complete 0.10 protocol spec, we have bumped the version to the 0.9.x series to indicate a redefined focus on stabilizing the current code base. You will notice significant API changes from 0.1.0 to 0.9.0. We hope to not have to make too many more changes during the current cycle of development and hope to get a 0.10 version out once we have completed coverage of the entire spec.
Some of the highlights of this release are:
Give me the code
About Kamaloka-js
Kamaloka-js is an implementation of the AMQP messaging protocol in native JavaScript. It is setup to be used with Orbited but can be used with any library which produce TCPSockets in the browser similar to Orbited. Kamaloka bindings are generated from qpid XML protocol description files.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]In my quest for cleaner code in Kamaloka-js I have been working on simplifying the dispatch model. AMQP has some interesting features built into it to facilitate real-time functionality along with message prioritization. To accomplish this messages can be sent on different queues and tracks, and also be broken up into segments which can be further broken up into frames.
Frames
Frames are the basic building blocks of the AMQP data stream. They contain complete headers that describe queue, track and segment that is currently being constructed. The payload of a frame (the segment being built) can be broken up into arbitrary sized byte arrays which are then reassembled based on the channel and track they are sent on. In this way applications with memory constraints can request that frames be no bigger than what the application can fit in memory. A typical frame header looks something like:
{
channel: 0,
track:1,
is_first_frame: 1,
is_last_frame: 1,
is_first_segment: 1,
is_last_segment: 1
}
Segments
Segments are like frames but instead of an arbitrary split on a data size, each segment is split on struct boundaries. That is to say, when you receive a complete segment you can be sure that it can be fully parsed. There are currently four segment types: A control segment, command segment, header segment and body segment. Command messages are currently the only type that can contain a header and body segment. For instance, the transfer command is used to send messages to and from a queue. It would contain header segments which could be used to route the entire message and it would contain a body segment which contains arbitrary data the user application cared about. Each segment is broken up into at least one frame. Multiple segments would never be sent in a single frame.
Channel and Tracks
A channel is just an integer that denotes related frames and segments. One can think of each channel being a list of incoming frames which are ordered correctly. Once the last frame in a channel is seen, the message is constructed and the channel is flushed and ready to receive a new message. In this way, multiple messages can be received at the same time by utilizing different channels but only one message can be sent on a single channel at a time.
Tracks are an exception to the one message per channel rule. There are two tracks in the current spec. The control track (track 0) and the command track (track 1). Controls preempt commands on a channel, so you can be in the middle of receiving frames on the command track and a control can come in on the control track and you must respond to that first.
This all sounds complicated but you can just think of the channel/track combination as being one entry in an hash. For instance frames coming into channel 0 and track 1 would be given the hash “0.1″:
message_channels["0.1"].add_frame(incoming)
First pass – Frame dispatching
At first it was easier to think of the frame and segment issues as different layers. At the lowest layer I would decode and dispatch each frame and then pass it off to the segment layer once a complete segment had been decoded. The segment layer would then collect the segments, relate them to each other, and then construct the full message. The frame layer looked something like this:
Flowchart showing the frame decoding layer of the kamaloka-js AMQP bindings
The dispatch would then pass it off to the segment decoder.
This became overly complicated because each segment had varying degrees of metadata and the body and header segments didn’t map to the message object very well. I could have gone ahead and created a segment object but I wanted to simplify the code.
Flattening the frame and segment layers
As it turned out flattening the model only added a couple of more steps to the current frame layer. Since frames and segments are just two different ways of breaking up a message for transfer over the wire, combining the two in the same layer made sense. What I ended up with was this:
Flowchart showing how the Kamaloka-js AMQP bindings decode frame and segments into a message
If you notice I now only create a new message if it is the first frame and first segment on a channel. When I see it is the last frame I incrementally decode the segment but I only dispatch the message once the last segment is seen. In the end, these minor adjustments allowed me to strip out a whole layer of redundant code. It also simplified the low level event code as I used to have to manage callbacks for each segment in order to construct a message. Now events only trigger once a full message is received and not when each frame or segment is received.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]Those who are following AMQP know that work on version 1.0 of the protocol spec is happening right now. Previous versions of the API were heavily dependent on the protocol itself but with 1.0 around the corner a new messaging API has come along to help bridge the gap between 0.10 and 1.0 for the most common use cases.
In kamaloka-js, the JavaScript AMQP bindings, I have been working on implementing this API along with cleaning up the codebase significantly. Today I put the final touches on multi-part frame decoding as well as the dispatching code and hope to have a brand new release next week. Here is a look at both the old and new API:
Old kamaloka-js API as used in release 0.1.0
<script src="/static/Orbited.js"></script>
<script src="/javascript/amqp.protocol.js"></script>
<script src="/javascript/amqp.protocol_0_10.js"></script>
<script src="/javascript/qpid_amqp.js"></script>
<script>
amqp_conn = new amqp.Connection({host: 'localhost',
port: 9000});
amqp_conn.start();
// You should have your server generate a UUID since browser methods
// are unreliable at best
session = amqp_conn.create_session('not_a_great_id' +
(new Date().getTime() +
Math.random()));
var remote_queue = settings.remote_queue + session.name;
session.Queue('declare', {queue: remote_queue});
var queue = session.create_local_queue({name: '0'});
var output_cb = function(msg) {
console.log(msg.header.delivery_properties.routing_key +
' sent ' + msg.body);
}
queue.subscribe({exchange: 'amq.topic',
remote_queue: remote_queue,
binding_key: 'com.j5live.#',
callback: output_cb});
queue.start();
// test a multi segment transfer
setTimeout(function(){
session.Message('transfer',{accept_mode: 1,
acquire_mode: 1,
destination: 'amq.topic',
_body: 'Test Transfer From Browser',
_header: {delivery_properties:{
routing_key:'com.j5live.test'
}
}
});
}, 10000);
</script>
New kamaloka-js API as used in the upcoming 0.9.1 release
<script src="/static/Orbited.js"></script>
<script src="/javascript/jsio/jsio.js"></script>
<script>
jsio("import qpid_amqp as amqp");
jsio("from amqp.protocol import register");
// load the 0.10 version of the protocol
register("amqp.protocol_0_10");
amqp_conn = new amqp.Connection({host: 'localhost',
port: 9000,
socket_cls: Orbited.TCPSocket
});
amqp_conn.start();
// You should have your server generate a UUID since browser methods
// are unreliable at best
session = amqp_conn.session('not_a_great_id' + (new Date().getTime() + Math.random()));
var output_cb = function() {
msg = this.fetch();
console.log(msg.get('_header').delivery_properties.routing_key +
' sent ' + msg.get('_body'));
}
var receiver = session.receiver('amq.topic/com.j5live.*');
receiver.onReady = output_cb;
receiver.capacity(0xFFFFFFFF);
// test a multi segment transfer
var sender = session.sender('amq.topic/com.j5live.test');
sender.send('Test Transfer From Browser');
</script>
The new API will offer a lot more control, better dispatching and a simplified interface. Further integration with js.io is planned as well as full support for transactions and flow control.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]If there is one thing I learned when working on D-Bus is that if you build a decent API, people will use it. Right now I am working on Kamaloka-js – JavaScript bindings for building JavaScript native AMQP clients. As AMQP starts moving towards 1.0 a new high level API, based loosely on JMS, is emerging to bridge the gap between 0.10 and 1.0. Previously with AMQP you pretty much worked with patterns on the protocol level which meant when the new specification came out you would pretty much have to rewrite to the new API. The new messaging based API seeks to decouple the protocol from the API for the most common use cases.
As I implement the JavaScript API there are some decisions I must make regarding how closely I stick to this new API as implemented in other bindings. Lets look at how the new API receives messages in Python:
...
conn = Connection.open(url.host, url.port)
ssn = conn.session()
rcv = ssn.receiver('amq.topic/org.j5live.demo')
while True:
try:
msg = rcv.fetch(timeout=timeout)
process_message(msg)
ssn.acknowledge()
except Empty:
break
except ReceiveError, e:
print e
break
If you notice the API is very much mainloop based. With every loop we fetch a message off the local queue. If there are no messages, fetch works very similar to the way poll does for sockets – sleep until woken up by network traffic or a timeout. The issue is if we did this in JavaScript we would block the UI and event propagation (newer browsers do expose threading but that feature can not be relied upon in all browsers). If you have ever mistakenly created an infinite loop in JavaScript you will understand how bad this is.
There are two options I see here, keep the fetch method and require the programmer to use timers to read from the local queue or, as currently implemented in my own high level API, attach callbacks to receivers.
The timer/fetch method has a couple of advantages. First it more closely resembles the API of the other bindings. Also it allows tighter management of the local queue. Since each time the timer is triggered we can determine how many messages we wish to process we can give some performance guarantees based on how long it takes to run operations in response to the messages. In this way we can dictate when a message is processed and tweak applications to run smoother. This comes at the price of added complexity which could be even more of a detriment to performance when in the wrong hands.
The callback API has the advantage of being more in line to what JavaScript developers expect. JavaScript is highly event/callback based with onFoo handlers everywhere. We currently work this way. When a message is decoded from the socket we dispatch it immediately to any callback registered to listen for it. The issue here is an application can get hammered with messages with no way to defer processing (except by implementing their own processing queue).
Why not implement both? This is an option but that doesn’t mean it is a good option. Multiple ways of doing the same thing often confuse new users. Understanding the differences between the API’s and the nuances between usecases are often more complex than either API alone. This could be daunting to a new user.
The trick is to make the API’s build on one another. For instance I could add an onReady handler which then allows the user to use the fetch API to grab any messages inside the handler. We would then set up an internal timer if the queue was not completely drained. This would require users be familiar with the fetch API without having to set up their own timer. If they wanted to have more control, they could set up their own timers instead of using the handler.
Any JavaScript developers have any insight as to the best way forward?
[read this post in: ar de es fr it ja ko pt ru zh-CN ]With FUDCon being only two weeks away, I’ve been polishing up my presentation and looking into what is in store for the future of Fedora’s Infrastructure. My main focus has been adding “push” capabilities throughout our infrastructure via the use of the AMQP protocol and qpid servers.
So why do we care about push messaging?
One can think of messaging as a conversation between two people. Poll messaging goes something like this:
Push on the other hand is a bit less chatty:
By eliminating the need to poll to know when an event has happened within Fedora’s Infrastucture, and making it easy for services to push out events in an easy fire and forget manner, we open up the door to a number of interesting possibilities. For instance, instead of implementing mail notification functionality in every service we simply create a mail notification service that listens for events and sends e-mails to people who what them. Do you want to get a notification on your desktop when your build is done instead? This makes it possible to provide that functionality without bogging down the Koji build service.
Human consumable notifications isn’t the only advantage of adding push messaging to our infrastructure. Because the data is first formatted for services to consume, notifications can be used for things like automation and synchronization. For instance someone could write a script that listens for new git checkins at fedorahosted.org and tries to package it into a private repo for personal testing.
What do we need to discuss at FUDCon?
Though this change is minimally invasive and you can ignore it if you don’t need to work with notifications, it is never the less a large undertaking. Some of the things we need to discuss are:
AMQP Sessions at FUDCon
There are a couple of talks planned and a hackfest. Come to the talks to get a deeper understanding of AMQP and how we envision using it within Fedora and then attend the hackfest to help us map out the future of the Fedora Messaging Infrastructure.
Saturday:
Sunday and/or Monday:
I am proud to announce version 0.1.0 of the kamaloka-js AMQP bindings. This initial release is considered to be an alpha quality release. It is there for you to break and give feedback so we can continue to improve the bindings and eventually get them to the point where they can be used to run mission critical applications.
In the next few days I will be concentrating on documenting the design of the bindings so anyone who wants to can easily jump in and contribute patches as well as suggestions. Please visit the project page to find out more and see how you can help out.
What’s in store for the future?
Where can I find this release?
What’s in this release?
How can I find out more?
Kamaloka-js is an implementation of the AMQP messaging protocol (http://amqp.org/) in native JavaScript released under the MIT license. It is setup to be used with Orbited (http://orbited.org/) but can be used with any library which produce TCPSockets in the browser similar to Orbited. Kamaloka bindings are generated from qpid XML protocol description files.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]While I am developing on Firefox today I tested and fixed some issues under Chromium (webkit based), Opera (opera based) and Konquorer (khtml based). My pub/sub demo worked the same in all four browsers (and at the same time I might add). We tried with IE8 on Luke’s machine but the error messages were cryptic. I’ll have to get my hands on a windows box at some point but I’m guessing the biggest issues are going to be trailing commas or keywords which are similar in the amqp spec and javascript (e.g. I ran into void and class keyword issues with the webkit browsers).
One thing to note is Chromium’s integrated debugger is a lot faster than Firebug. I haven’t used it extensively but if it can break at breakpoints and introspect objects without getting screwy I might end up switching. I really like firebug but lately it has become dog slow. It usually takes a couple of minutes of waiting for each frame of data to be processed and displayed while under Chromium with their debugger on, the slowdown is noticeable but not significant compared to when the debugger is off.
UPDATE: Chromium’s debugger is probably faster because it doesn’t actually do anything that I can see. It displays scripts but doesn’t actually break into the code
[read this post in: ar de es fr it ja ko pt ru zh-CN ]Thinking Outside The Box: Bringing the Network back into G(N)OME
Thanks to Paul Cutler’s excellent work on GNOME Journal we have a great stream of articles to keep us up to date on the happening of GNOME. Please consider helping him out if you have time on your hands. It is a great way for those who don’t program but still love GNOME to become a producer instead of just a consumer in the GNOME ecosystem. The potential for making an impact with contributions such as these is just as important as those who contribute with code.
On the heels of Paul’s interview with myself I thought I should post the slides of my talk. Unfortunately the video of my talk did not seem to make it with the numerous other published videos. I’m not sure why that is but it is something we should improve on next year. Being able to reach those who could not attend the conference is just as important as presenting to those in the conference hall.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]I managed to make it home in one piece from Vegas and dove right back into work. After getting away from messaging for a bit I have come full circle but with a little bit of a twist. In an effort to bring some of the goodness that is interprocess messaging to the interwebs – similar to what we get on the desktop with D-Bus – I am implementing the standard AMQP protocol in native JavaScript.
This along with QMF will give us the ability to send out notifications of events, monitor statistics and eventually supplement REST and XML-RPC interfaces with more efficient multi-response method calling (imagine streaming rows of data instead of waiting for the entire row set to be sent).
AMQP Handshake over Orbited
Above is a little hex dump web app I wrote to test the bindings. What you see is the start of the AMQP handshake. First I send the header indicating the version of the protocol I am expecting. I then get the ’start’ response back stating the capabilities of the broker I have connected to. I then send the ’start-ok’ packet indicating what auth mechanisms I support and the locale (I would also send a token if I were using a particular auth other than anonymous).
That is about as far as I have gotten but since the bindings are autogenerated from an XML description of the protocol most of the control and command classes are implemented. There are still some types I am trying to figure out how to support (the supported types are a magnitude above what was needed for D-Bus) and I need to smooth out the rough edges as well as write the higher level user bindings but I am off to a good start.
Code will be posted as soon as the project resources are approved and setup. Keep watching my blog for updates.
BTW: The slow response times in the time stamps are due to running inside of the firebug debugger and writing out the hex dumps on the fly using JQuery. It seems the more sophisticated JQuery and Firebug get, the more they butt heads.
[read this post in: ar de es fr it ja ko pt ru zh-CN ]Powered by WordPress