J5’s Blog

I haven’t blogged lately. For some reason since the popularization of Facebook posts and tweets, my ability to write more than a few coherent sentences have greatly diminished. Perhaps it is just me getting old but change is what change does and a lot of change has happened recently. The biggest recent change is me getting a promotion to Senior Software Engineer and moving from the Fedora team to the OpenShift team inside of Red Hat. Yes, I have traded Beefy Miracles for Space Pandas and I think the change has done me some good. I have wanted to transition to a more customer driven structured part of Red Hat without sacrificing the excitement of working with a fast moving project. OpenShift fit the bill very nicely with their agile development workflow in the emerging field of PaaS (Platform as a Service) cloud development. It is also nice having a large and growing team to work with.

My involvement with PyGObject

That being said most of my hacking time will be spent on OpenShift related projects and while I had already transitioned out of day to day PyGObject maintainership some time ago, I will no longer have any real time to dedicate to the project (I’m actually learning Ruby right now). To tell the truth, not being able to put any more serious time into the project is one of the major reasons I decided I needed a change. There are a number of other people still contributing to the project but it is sorely in need of a lead maintainer who can do releases, keep people on schedule and ping the right people when bugs languish. I feel PyGObject is in good shape but as it begins to get more uptake bugs fixes need to be committed, edge cases corralled and the last mile needs to be traversed. I will still hang out in #python on GIMPNet and can be persuaded to look at patches or even write a few if you ping me and are nice.

Jobs

With me leaving the Fedora team there is now an opening for someone to join the team. They are looking for an all around FOSS rock star who can work in a number of different areas such as packaging, desktop and web development, and any number of miscellaneous skill your would encounter with any FOSS project. The main responsibilities would be maintaining, improving and integrating our infrastructure tools such as Fedora Community Packages web app, Bodhi update tool and Fas accounts system as well as developing tools to make it easier to contribute to Fedora. Most of the tools are written in Python so being a Python expert is a big plus as well as having worked as part of a team on any major open source project. If that sounds like fun to you send me your resume (I get a bonus if you get hired).

OpenShift is also expanding so if any of these jobs look more like your speed feel free to mail me also.

PyGObject 3.0 has been released!!! This is the stable release we have all been waiting for.  3.0 has been stabilizing for some time.  This marks a huge improvement over the 2.28 branch in terms of speed, stability and GObject API coverage.  With  this release PyGObject remains the most complete language bindings for developing application with GObject Introspection.

I want to thank everyone who has been a part of this release, from those who reported bugs to those who submitted large patches, and also those who cheered us on from the sidelines or gave us support in other areas.  At this point in the project they are just too many to list so I won’t, in fear that I may accidentally leave people out.

Highlights Since 2.28

• New rewritten invoker is twice as fast and easier to extend and debug
• Complete break from static bindings so we may improve the core without breaking legacy APIs
• Better type handling
• Parallel installable with PyGObject 2.28 for legacy binding support (2.28 must be compiled with –disable-introspection)
• Support for function calling using keyword arguments
• Support for multiple arrays referencing a single length parameter (e.g. Clutter.Actor.animatev)
• Once again we support Windows builds with some caveats

Going Forward

As we move into another unstable cycle there are a number of goals we will strive for the next major release.

• Documentation – we jump started this at the Desktop Summit but a lot of work is needed to get this to the quality currently exhibited by the PyGTK bindings.
• Full Gtk subclassing – We are almost there but we still need support for calling C callbacks in Python to support overriding interfaces such as Gtk.Widget.for_all()
• Getting rid of all legacy marshallers – currently they are still being used for marshalling vfuncs and signals
• Cutting out the middle man – We should get rid of our dependency on GArguments and marshal directly to FFI
• Moving away from a Gtk+ focus – Clutter and GStreamer 1.0 are just as important to fully support
• Full Gio support – right now dealing with byte streams isn’t completely introspectable

ChangeLog

http://download.gnome.org/sources/pygobject/3.0/pygobject-3.0.0.changes  (1.34K)

Download

http://download.gnome.org/sources/pygobject/3.0/pygobject-3.0.0.tar.xz  (530K)
  sha256sum: ef6735792b0d44287126a6a3b181c85559849063d770506fe06848adb87ce815

http://download.gnome.org/sources/pygobject/3.0/pygobject-3.0.0.tar.bz2 (629K)
  sha256sum: 5682603c58be67e336e8cfaf19d6321158425797a7da12f646cc8706508ab95c

About PyGObject

GObject is a object system used by GTK+, GStreamer and other libraries.

PyGObject provides a convenient wrapper for use in Python programs when accessing GObject libraries.

Like the GObject library itself PyGObject is licensed under the GNU LGPL, so is suitable for use in both free software and proprietary applications. It is already in use in many applications ranging from small single purpose scripts up to large full featured applications.

PyGObject now dynamically accesses any GObject libraries that uses GObject Introspection. It replaces the need for separate modules such as PyGTK, GIO and python-gnome to build a full GNOME 3.0 application. Once new functionality is added to gobject library it is instantly available as a Python API without the need for intermediate Python glue.

PyGObject 2.90.4 has been released. This is an unstable release leading up to the PyGObject 3.0 release coming soon.

Highlights

• Various deprecated API that was only needed for the static bindings have been removed which means you must recompile anything that links to PyGObject (currently libpeas and glade)
• Overrides directory variable was taken out of pygobject-3.0.pc and moved to the gi._overridesdir attribute to avoid having multiple .pc files for each version of Python.  This means third parties who install overrides need to import gi from their install scripts with the version of python they are installing for.
• Multiple arrays referencing a single length parameter are now supported along with flat GValue arrays.  This allows APIs such as Clutter.Actor.animatev to be bindable.
• Refcount crasher bug triggered when using GObject.new was fixed so glade can now import custom Python  GtkWidgets.
• Build system now works with MinGW environment in Windows.
• Python 3 now checks instance types again.
• Documents disabled since they aren’t useful yet and present parallel install issues.
• Demos were fixed up to better reflect the preferred way of using PyGObject.

ChangeLog

http://download.gnome.org/sources/pygobject/2.90/pygobject-2.90.4.changes  (20.2K)

Download

http://download.gnome.org/sources/pygobject/2.90/pygobject-2.90.4.tar.xz  (529K)
  sha256sum: 8407b6997181bbca4783798e21d7d63ca41708a6c05a3b08c953d64e7b97b2a1

http://download.gnome.org/sources/pygobject/2.90/pygobject-2.90.4.tar.bz2 (629K)
  sha256sum: 467eb39f06664fb8bbb9ab407a15b8d56a63a92ac2f0914afc8518aedd765c43

About PyGObject

GObject is a object system used by GTK+, GStreamer and other libraries.

PyGObject provides a convenient wrapper for use in Python programs when accessing GObject libraries.

Like the GObject library itself PyGObject is licensed under the GNU LGPL, so is suitable for use in both free software and proprietary applications. It is already in use in many applications ranging from small single purpose scripts up to large full featured applications.

PyGObject now dynamically accesses any GObject libraries that uses GObject Introspection. It replaces the need for separate modules such as PyGTK, GIO and python-gnome to build a full GNOME 3.0 application. Once new functionality is added to gobject library it is instantly available as a Python API without the need for intermediate Python glue.

I’ve been tracking down some crashers and refcount issues as well as cleaning up the Python 3 support.  Next release will also remove all of the file conflicts associated with PyGObject-2.28 parallel installs.  We also have support for compiling under windows using MinGW thanks to Dieter Verfaillie. If you have a bug you care about now is the time to ping me in #python on irc.gnome.org.

One of the big topics of during the GObject Introspection Hackfest in Berlin has been generating documents directly from the .gir files.  While we are far from having complete documents for every supported language binding but thanks to Tomeu Vizoso,  Shaun McCance, Laszlo Pandy and Colin Walters with myself holding up rear after my work on the PyGObject 3.0 pre-release, we have a working framework.  Examples of the first output are available on my peoples page and you can get more information on the project page.

I would like to thank the Gnome Foundation for sponsoring travel and hotel as well as space during the Desktop Summit; Openisums for supplying us with space after the Summit was over; Nemein for the excellent Fondue dinner; Collabora for the Africana food that I had to miss due to the equally excellent cocktail party at Kat and Dave’s place; and KDE for the yummy Vietnamese lunch.

Much has been sewn and much has been grown, but now it is time to go.  Another successful hackfest is in the books and now the hard work of building on our achievements begins.  But first, some beer.

I am pleased to announce version 2.90.1 of the Python bindings for GObject. This is an unstable release of what will eventually become PyGObject 3.0. PyGObject 3.0 is the first release that is completely based off of Introspection. To support legacy static bindings like PyGtk this release is parallel installable with PyGObject 2.28 provided introspection is turned off in the older module.

The new release is available from ftp.gnome.org:

http://download.gnome.org/sources/pygobject/2.90/

What are the highlight of changes since PyGObject 2.28?

• New rewritten invoker is twice as fast and easier to extend and debug
• Complete break from static bindings so we may improve the core without breaking legacy APIs
• Better type handling
• Parallel installable with PyGObject 2.28 for legacy binding support (2.28 must be compiled with –disable-introspection)
• Fixed object array handling
• Added more overrides for PyGTK API emulation
• Support for function calling using keyword arguments
• GObject and GLib symbols can now have overrides
• All static bit removed or made private
• GVariants now work from callback returns

Do I need to port my apps from PyGObject 2.28 to PyGObject 3.0?

We tried to keep the API as close to what we were recommending for 2.28 however since 2.28 straddled both the static and dynamic APIs there are a few things to keep in mind when running under 3.0:

• don’t import pygtk – this was for static bindings only and while it may have been harmless in 2.28 it will cause conflicts in 3.0
• import gobject must be ported as from gi.repository import GObject – this was the recommended way of importing GObject but some code may still have used import gobject.  Having both will cause applications to fail because it will load the older 2.28 bindings alongside the 3.0 bindings.
• import glib must now be ported as from gi.repository import GLib – this is a new requirement and prevents the issues discussed above
• use the gi.require_version API (e.g. gi.require_version(’Gtk’, ‘3.0′)) – This is more of a recommendation as we will be outputting warnings in the future.  Due to the rapid development of certain libraries like the upcoming Gtk-4.0, it is best to specify the major version of the library you wish to target instead of relying on PyGObject to select the latest one available on the system.  This also gives helpful errors if the correct version is not installed on the user’s system.

Blurb:

GObject is a object system used by GTK+,  GStreamer and other libraries.

PyGObject provides a convenient wrapper for use in Python programs when accessing GObject libraries.

Like the GObject library itself PyGObject is licensed under the GNU LGPL, so is suitable for use in both free software and proprietary applications. It is already in use in many applications ranging from small single purpose scripts up to large full featured applications.

PyGObject now dynamically accesses any GObject libraries that uses GObject Introspection. It replaces the need for separate modules such as PyGTK, GIO and python-gnome to build a full GNOME 3.0 application. Once new functionality is added to gobject library it is instantly available as a Python API without the need for intermediate Python glue.

Introspection/Python 2 bindings requires gobject-introspection >= 0.9.5 and py2cairo >=1.10.0
Introspection/Python 3 bindings requires gobject-introspection >= 0.9.5, pycairo >=1.10.0 and Python >= 3.1

–
John (J5) Palmieri
GNOME Foundation member
johnp@redhat.com

And I am very tasty. Though I missed the main portion of the Desktop Summit I will be in Berlin early tomorrow (Wednesday) morning. I should be in time for the Introspection BOF provided I don’t get lost. I trust that taking the TXL bus to the Staatsoper stop will prove to be uneventful.

Tomeu finished up his review this weekend and gave me the green light to merge my long awaited invoke-rewrite branch. Besides being twice as fast as the previous invoke implementation, this branch cleaned up the code into discreet layers and modules which should make it safer and easier to debug as well as profile and add more speed improvements. I have also beefed up memory cleanup in the case that a call fails and the parameters have yet to be marshalled to python objects.

This rewrite is the basis for the future PyGObject 3 release. PyGObject 3 will permanently remove the static bindings but be parallel installable with PyGObject 2 provided that introspection is turned off in the PyGObject 2 build. While this may cause a small amount of pain for packagers, this will provide a smooth upgrade path for developers with little to no expected change to their current PyGObject Introspection projects.

As soon as we can make the two parallel installable we will do a release, update the JHBuild module sets and get the word out that unstable distros should switch to using the new setup. We are committed to stabilizing PyGObject 3 for the GNOME 3.2 release.

I just filed a bug to request a review for merging my invoke-rewrite branch. This is a huge overhaul of internals of the introspection layer of PyGObject which I feel both speeds up calls as well as makes it easier to fix bugs and add features.

Right now the invoke-rewrite branch is stable and complete enough to be merged to master. It passes all the tests and even cleans up after itself when an error is raised while processing input parameters (something the current code doesn’t do).

A brief list of the advantages of this branch

• over 2 times faster running thought the test suite than the current implementation.
• more robust – handles a couple of corner cases which the current code does not such as multiple lists pointing to the same aux length value, having argument 0 be a length value and correctly blocking all NULL values which are not marked as allow-none
• correctly handles argument parameter counting and reduces the amount of complicated math used to determine argument counts and indexes
• splits code into logical layers such as caching, invoking, marshalling and cleanup layers
• splits code into manageable units so, for example, if there is an error in list processing it is easier to set break points and focus on the list code instead of wading through large switch statements
• ripe for even more optimizations and better layout for profiling

Things that are yet to be done

• Memory management still leaks and some types such as gclosures are not yet cleaned
• Some of the earlier code like struct marshalling can be split up further
• signals and closures still use the old marshalling paths
• remove the FFI/GArgument layer and marshal directly to FFI

Why merge now?

Code rots and getting more people to use the new codepath makes it easier to fix bugs. It is already feature complete with the old code and simply trades off some new bugs while fixing a slew of other bugs. Because of the caching layer the code is somewhat more complex but follows a much more consistent pattern. In my eyes it is easier to understand once learned but developers need ramp up time since the structure is different. This is the future, there is no need to keep bandaging the old invoke code. As we get to more and more corner cases it gets harder to patch the old code without breaking other pieces or causing memory leaks that are hard to track down and plug (see garray hack we added to handle C Array vs GArray failures).

A brief overrview of the design

New modules

• pygi-invoke-ng – the new invoke API handles setup/invoking/teardown at a high level. Replaces pygi-invoke and does not attempt to handle memory it did not itself create (see garray handling for example)
• pygi-invoke-state-structure – the state struct used to pass around changing state during the caching, marshalling, invoking and cleanup stages
• pygi-cache – sets up the cache which normalizes the information and support methods such as marshalling and cleanup of each interface when called. This allows us to process method calls in a more efficent manner as well as makes it easier to debug
• pygi-marshal – marshalling functions for each parameter type. Replaces pygi-arguments
• pygi-marshal-cleanup – cleanup functions for each parameter type.  Includes cleanup for parameters which were not processed due to an error.

Basic flow

invoke is called on an GICallableInfo object
  |
  |-> if GICallableInfo does not have a cache we loop over the interface
  |    creating the PyGICallableCache and ArgCaches for the interface
  |    and its arguments
  |
  |----> use the cache to marshal the parameters into the state struct
  |
  |---------> using the marshalled values invoke the interface using the
  |           GI FFI calls
  |
  |----> use the cache to cleanup any in values that need cleaning
  |
  |---------> use the cache to marshal the out values to python objects
  |
  |----> use the cache to cleanup any out values that need cleaning
  |
  |-> cleanup state
  |
return out values to python app

Standard paterns

All marshalling and cleanup methods for types are added to the argument cache as function pointers. A PyGIArgCache has these four function pointers no matter what the type we are caching for:

• in_marshaller – the function which marshals in values for this argument
• out_marshaller – the function which marshals out values for this argument
• in_cleanup – the function that cleans up in values for this argument
• out_cleanup – the function that cleans up out and in/out values for this argument

All type marshallers and cleanup methods follow this naming pattern:

• in marshallers – _pygi_marshal_in_<type> (e.g. _pygi_marshal_in_utf8 or _pygi_marshal_in_interface_enum)
• out marshallers – _pygi_marshal_out_<type> (e.g. _pygi_marshal_out_utf8 or _pygi_marshal_out_interface_enum)
• in_cleanup – _pygi_marshal_cleanup_in_<type> (e.g. _pygi_marshal_cleanup_in_utf8)
• out_cleanup – _pygi_marshal_cleanup_out_<type> (e.g. _pygi_marshal_cleanup_out_utf8)

These are all setup during the caching phase in functions named similarly (e.g. _arg_cache_in_utf8_setup)

I am pleased to announce version 2.28.3 of the Python bindings for GObject. There was an issue with the patch that went out with 2.28.2 and all users should upgrade from 2.28.2 to this release. This is a minor bugfix release of the stable 2.28 branch.

The new release is available from ftp.gnome.org:

http://download.gnome.org/sources/pygobject/2.28/

What’s new since PyGObject 2.28.1?

• fixed an ABI break in the static bindings when setting string gvalues
  • e.g. passing an int to a Gtk.ListStore column which expects a string automatically converts the int to a string instead of throwing an error
• This release fixes a bug that the 2.28.2 release introduced

What’s new since PyGObject 2.28.0?

• pygi-convert.sh now supports webkit conversions and favors using GObject over gobject
• Raw closures can now be passed from a signal/vfunc callback to a method
• Revert linking to the python libs because the python runtime statically links it in
• TreeModel column marshalling is now more robust (supports GObject Python Object storing)
• Gtk.MessageDialog now respects the MessageType
• You can now send None in for the signature of GDBus messages that have no parameters
• TreeViewColumn.set_cell_data_func can take None for the func_data
• Fix syntax error so we can run in Python 2.5
• Add pickers and menu demos

Blurb:

GObject is a object system library used by GTK+ and GStreamer.

PyGObject provides a convenient wrapper for the GObject library for use in Python programs, and takes care of many of the boring details such as managing memory and type casting. When combined with PyGTK, PyORBit and gnome-python, it can be used to write full featured Gnome 2 applications.

Like the GObject library itself PyGObject is licensed under the GNU LGPL, so is suitable for use in both free software and proprietary applications. It is already in use in many applications ranging from small single purpose scripts up to large full featured applications.

PyGObject requires glib >= 2.22.4 and Python >= 2.5.1 to build.
GIO bindings require glib >= 2.22.4.

The Introspection module is the next generation Python GObject library bindings. Instead of statically wrapping every GObject based library we can now dynamically accesses any of those libraries using GObject Introspection. It replaces the need for separate modules such as PyGTK, GIO and python-gnome to build a full GNOME 3.0 application. Once new functionality is added to gobject library it is instantly available as a Python API without the need for an intermediate Python module.

Introspection/Python 2 bindings requires gobject-introspection >= 0.9.5 and pycairo >=1.0.2 or py2cairo >=1.8.10
Introspection/Python 3 bindings requires gobject-introspection >= 0.9.5, pycairo >=1.8.10 and Python >= 3.1

–
John (J5) Palmieri
GNOME Foundation member
johnp@redhat.com