J5’s Blog

I’m sitting here at my parents house (I wanted to beat the snow for the holidays) working on the document that will guide the development of the new invoke module for PyGI. To start I attacked it from a very high level, idealistic standpoint where we could both modularize and optimize the code all in one sweeping iteration over the GI function arguments.

Of course I knew there would be edge cases and a few places where the model needed to be tweaked. As I dived more and more into the details (I’m designing the caching layer right now) I realized there isn’t enough information in GI to completely get rid of processing my own metadata, nor are the args guaranteed to be able to be processed linearly. For instance most library init functions take argc (array length) before argv (the array). Of course this isn’t to say that GI and the typelibs are not complete. They are more than complete, just that we haven’t normalized the state machine.

This all necessitates a metadata sorting loop which begs the question, why don’t we just combine that with the cache and state initialization layers and then just loop over the in args during the in validation and marshalling layer? We could for a majority of in arguments, process them in the metadata loop but for the few edge cases that break the model, it would mean separate code paths and increased complexity.

The truth is even though paper is entitled Speeding Up Invoke, my real goals are to fully understand what was going on under the hood while explaining it to others and to modularize the code so it is easier to understand (and optimize) going forward. In any case almost anything we do will be an improvement over the multiple times we call into the GI library for the same data in one invoke call. By producing a roadmap and breaking the code into smaller chunks, more people will be able to look at the code and provide optimization, feature and stability fixes. Actually the way things are laid out in the paper we will already fix some of the limitations such as simplified argument counting which allow optional user_data values, multiple callback support, and multiple lists/arrays referencing one length in value (see clutter_actor_animatev for an example).

The good news is once I fully flesh out the paper implementing it should take a couple of days. Most of it is just rearranging code. I hope to get to it before the hackfest in the middle of January.

Download: PyGI speeding up invoke 0_1 draft.pdf – this is and highly incomplete first draft of the paper.  It lays out my design goals as well as explaining the invoke flow and getting down to the details of in value validation.  Comments and constructive criticism welcome.

I’ve been taking  a break from coding to really map out the internals of invoking a function using GObject Introspection.  A few bugs popped up that I could either band-aid or look into really fixing some of the core issues we have in our invoke routine.  I wanted to get a deep understanding of how invoke worked and how all of the GI function fit together to validate and marshal between C and Python.  While I am at it I am also designing an idealistic view of how we could modularize the validation and marshalling routines so that it is easier to experiment with optimizations and add support for other types should they be needed in the future.  I say this is an idealistic view because there are still going to be limitations and edge cases we will have to work around but it is my hope that by writing up a design and cleaning up the code will make it easier for others to understand the GI internals and contribute to the code.  While my work is geared towards the Python GObject Introspection bindings, I hope it will help other language binding authors better understand the GI internals.

When I started hacking on PyGObject Introspection I was using custom jhbuild module sets from the gnome-shell project.  This made it hard to tell people how to set up their own build roots.  With the new GNOME module reorg I decided to try out stock jhbuild and rebuild my entire build root.  I am happy to say that the process was really easy.

• follow the instructions on setting up jhbuild
• when editing your .jhbuildrc change the moduleset to point to gnome-world-3.0
• run the command jhbuild build pygobject

You may also want to build these libraries so you have something to work with:

• gtk+
• gstreamer, gst-plugins-good, gst-plugins-bad
• clutter, clutter-gtk
• WebKit

Many thanks goes out to those who have been involved with jhbuild and the GNOME release team for making it super easy to run and test GNOME modules throughout the years.