J5’s Blog

After sweating a bit and then Colin Walter’s wonderful caller-allocates patch being committed to gobject-introspection, I give you PyGI’s gtk-demo clone:

A clone of gtk-demo

It may not look like much but consider this, the demo uses a TreeView, TreeStore, ScrolledWindow, TextView, TextBuffer, TextTags and numerous types of Iters without asserting inside of GTK (provided you have the latest GTK from git). I can also now drop a python file in a directory and have it show up in the demo list which means many little snippets to methodically run through the gamut of the GTK API and fix any issues. Right now I need to write unit tests for the patches that enable gtk-demo.py to run but once I have those in I will commit gtk-demo.py to the PyGI repo.

Progress is sweet but hard work which is why I will be away this long Memorial Day weekend, sans any electronic devices, hiking the Glastonbury Wilderness. I’ll be back hacking on PyGI this Tuesday.

Well it has been awhile since we had the awesomely productive GNOME Python hackfest and now we have a roadmap and people working on getting out releases of the various components. At first I came in as a concerned citizen who just was just a consumer of the Python stack. I thought I would just organize a hackfest, attend a couple of days and everything would work itself out. I find naiveté helpful when getting started in a project because if I walked in knowing the amount of work involved I would start whistling, make a sharp U turn and hope nobody noticed me running for the hills.

As it stands I am lucky enough to have a Manager who realizes the importance of the GNOME Python stack (given that a lot of Red Hat’s tools including the installer are based off of them). Fedora 13 already has a parallel installable Python 3.1 stack and since we are targeting Fedora 14 to get as many major modules ported as possible, I have been give permission to work full time on the GNOME bits.

So what exactly have I been up to?

I’ve been working on porting PyGTK apps to PyGI with an eye towards fixing up the PyGI API through the overrides, finding and fixing bugs and adding binding friendly API’s directly to GTK+ (as well as fixing up some of the introspection annotations).

Where am I at?

I am happy to say that yesterday I managed to squash what seems to be the last of the memory crasher bugs that have been frustrating my API fixing efforts. They still need to either be vetted or are waiting on other patches to be incorporated into git so for now you can apply the patches in bugs 617780, 618049, 619007 and 583909. Actually the last two patches are redundant, the latter being the correct patch and the former being the patch we will go with if pygobject take some time to approve. PyGObject is a lot more risk averse since the PyGTK static bindings also relies on its API being stable.

I am also pretty sure that most of the GtkBuilder/GtkUIManager stack is now usable with minimal porting efforts from PyGTK. I should be pushing most of my override work into git fairly soon (I need to write test cases). One issue is they rely on patches to GTK such as the one in bug 61857 which won’t be in GTK until the 3.0 release. This leads to another issue of finding all the needed API changes. Since GTK is on a 9-12 month release cycle, not to mention API freeze dates, it might be hard to find all the API that needs adding in order to be binding friendly. Bellow I will talk about some strategies to deal with this.

What needs to be done and where can others help?

The biggest hurdle to making PyGI usable is gaps in API coverage. Because PyGI’s API is mostly auto-generated on the fly it doesn’t always get things right or throws in a lot of C’isms which are confusing in a Python world. Some APIs just plain can’t be wrapped correctly and will lead to refcounting errors and ultimately crashes. PyGTK on the other hand has been developed for many years and each API has evolved to fit within the Python world. Some are APIs better wrapped than others but I digress. What needs to be done is port small PyGTK examples that utilized different parts of the GTK+ API. For things that don’t work we need to file bugs and write overrides for those pieces of API. We are not aiming for a 1 to 1 port of PyGTK. For instance PyGI’s handling of constants, enums and flags is much nicer and more complete than PyGTK. What we are aiming for is to make it easy enough to port applications and to be able to fully utilize the GTK API.

As for patches to GTK goes I think we should have a two pronged approach. We should create a compat library that adds the needed API fixes but prefer API inside of GTK itself. The compat library will be for internal override use only and anything going into the compat library must first have a bug and patch filed against GTK. As we get a larger coverage inside of GTK, all new GTK APIs are vetted for scritability issues and distributions start removing support for older versions of GTK we can drop the compat library.

We also need people to read and vet patches as well as usher them through the approval process. Getting patches through quickly to meet schedules is very important. Helping to write tests for patches will also speed up the approval process.

Do I see any roadblocks in the way?

My number one biggest concern is the Tree and List model/view API. It is packed with C’isms so requires a lot of hand holding to wrap correctly. Also PyGTK has a rich, completely custom, tree API because of this. That means for the most part we have to hand write and maintain a lot of code in order to get the API into a usable state. I already took a crack at it but got frustrated when trying to wrap parts that used iters. It is all doable but in the end even PyGTK had to make some hard decisions (using TreeIters will almost certainly cause a leak). Does anyone want to write a new script friendly Tree API? I find this issue to be the dragon standing in the way of getting PyGI to a usable state. Let’s just hope it doesn’t turn into a windmill.

Where can someone ask questions and offer to help?

Right now the best place is #pygi on GIMPNet/irc.gnome.org. We are in the process of creating a mailing list and will inform everyone once it is up.

What a week. Get six smart people in the same room together, spend a little bit of money to get them there and get them comfortable, and good things happen. An in-depth praise for the hackfest process, which the GNOME Foundation Board has been putting more and more resources into every year, will have to wait until another blog post. Right now I want to thank our sponsors and quickly recap what was accomplished. First out sponsors, who made this possible:

• The GNOME Foundation board for providing the framework for the hackfest and travel assistance to Tomeu
• OLPC for providing us free space and all the water we could drink at their office in Cambridge.
• Canonical for making sure we were awake after the social events by providing us with coffee
• Red Hat, for feeding us with a nice Portuguese meal which we shared with the D-Conf/GSettings hackfest guys who were sponsored by Novell
• Myself, for sponsoring a couple of after hours social events to keep us all sane and allow us to discuss the future in a more social environment

I would also like to thank Walter Bender for helping us find a venue and Jeorge Castro for being the liaison between us and the Foundation. He is off to a great start as the newest Board Member.

Now to the meat of what was accomplished:

• PyGI saw its first formal release
• We suckered relative new comer Zack Goldberg into ongoing maintainership of PyGI
• Cairo, callback and virtual function support was added to PyGI
• pygobject and pygi both sprouted py3k branches on GNOME’s git servers which both fully compile and pass their unit tests (which probably means we need more unit tests). We aren’t going to move these to master for some time. But if you grab the branches and test them out the process will be much quicker.
• We were written up in Ars Technica

Right now I am porting D-Feet to use PyGI and will be testing out my D-Bus Python py3k branch after I get that up and running. D-Feet is a good test because it uses the GenericTreeModel from PyGtk as well as GtkBuilder elements. In both cases I have found places where I have had to add overrides to PyGI to complete the bindings. For instance, in the Builder I need to override gtk_builder_connect_signals which in C searches for C symbols which match symbols in the XML description file. This is useless in Python so I need to modify it to work the same way PyGTK works. Namely, by being able to pass in a dictionary or object with name/python function mappings (e.g. {’on_click’: on_click_handler}). It is not all that hard, especially since PyGI overrides are written in Python and not in C like PyGtk overrides. However I do have to get the exception behaviour correct so that might take some time.

All and all we are in excellent shape so start porting your apps and file bugs!!!

We are back from dinner where we met up with some of the D-Conf hackfest guys. Thanks to sponsorship from Red Hat and Novell we nourished ourselves with some decent Portuguese cuisine and are now back hacking for a bit before we turn in for the night. We’ve been working hard today, building on the great start we had on day 1.

Because of our work we have pushed py3k branches for both pygobject and pygi into the main GNOME git repo. They don’t pass their test suites yet but they compile under Python 2.6 and 3.1. The plan is to get them passing their test suites on these platforms and then test for regressions for older versions of Python.

Also the cairo bindings and callback support are almost ready to be merged into master pending code review. As soon as that happens I am going to merge it with the py3k branch and eventually we will move the py3k branch into master after much testing from the community.

I’m going to finish up debugging this long vs. int issue and will be heading home.

As day 1 is coming to a close let us quickly recount the work that has been started:

• We started by debating the way forward and came up with an action plan – In the short term PyGtk will be ported to Python 3.0 but start moving to using PyGI under the hood when wrapping non-overridden functions. PyGI will be developed in parallel to both support PyGtk and eventually replace it.
• I have gone and setup a build environment and documented it on the wiki along with the places to get various modules as well as helping out with some of the Py3k issues
• John Ehresman has been working on his github branch to bring Py3k support to PyGObject
• David Malcolm has been working on the PyGI Py3k support
• Zach Goldberg and Colin Walters have been working on the PyGI callback support, both code reviewing the current patches and discussing the best way to support callbacks given the way the gjs bindings handle them
• Tomue is working on introspection for cairo so we can use it from PyGI

We have arrived at One Cambridge Centre and the GNOME Python Hackfest is up and running. Thanks to the coffee sponsored by our friends over at Canonical, we are wasting no time whipping these bindings into shape.

If you wish to participate you can follow us on #pygi on gimpnet and look at the wiki at http://live.gnome.org/Hackfests/Python2010.

I’m getting my jhbuild environment up and running and will post instructions to the wiki as well as updates my blog throughout the day. For now here are the module you will need to help us hacking. The wiki also has a list of bugs with patches to check out.

• glib – from your distro
• gobject-introspect – from your distro or git head
• gir-repository – from your distro or git head
• pygobject – git head
• pygi – git head
• python3 – from python.org or your distro (optional if you want to help with the python3 port)

All of the above modules besides python3 can be found at the gnome.org git repos.

As we are getting ready to host the GNOME Python Hackfest I would like to take this time to thank our sponsors. Thanks goes out to The GNOME Foundation for sponsoring Tomeu’s travel; One Laptop Per Child for giving us a place to hold the hackfest; to Canonical for supplying coffee and some snacks to keep us awake; and to Red Hat for sponsoring a dinner for us hungry, hungry, hackers.

I will also be personally sponsoring tickets to a concert featuring Carbon Leaf and local act Tim Blane. I’m actually in it for Tim Blane who’s latest EP Traces is available for free in Vorbis and Flac formats. Yay for free music in free formats!

For those of you who haven’t seen Tomeu Vizoso’s blog on the hackfest we are trying to pull together this is a second call to PyGObject, Python 3 and GtkIntrospect hackers who might want to join us in getting the future of the Python bindings to GTK sorted out.

At issue are a couple of roadblocks to the continual maintenance of the the bindings. First is the lack of support for Python 3.0 and second is the unclear picture of how GNOME 3 effects us. The current plan is to finish the work done on PyGI (Python GObjectIntrospect support for PyGObject), at the same time making sure it all works under both Python 2.x and Python 3.x. Moving to the PyGI bindings should make the maintenance burden somewhat lighter for our busy maintainers.

If you have something you can contribute and are interested in attending our hackfest please take a look at the wikipage and e-mail myself or Tomeu. While this is being sponsored by the GNOME Foundation, you don’t need to be a Foundation member to be considered for an invite. We hope to be finalizing things soon and getting a budget to send to the board for approval.

Well, I’ve gone and done it. Thanks to David Malcolm’s excellent 2to3c tool and some hand wrangling with PyUnicode objects I was able to get D-Bus Python compiling and working on Python 3. Grab the patch and start testing it out.

I’ve also tested this under Python 2.6 but it would be nice to see if it also works under Python versions < 2.6 since 2.6 has a couple of compatibility layers built in.