J5’s Blog

I would like to thank everyone who came and hacked at the PyGObject hackfest, thank the GNOME Foundation Board for supporting us, Collabora for sponsoring us and Brmlab for hosting us. I would also like to give a big thanks to Tomeu for showing us true Prague hospitality (besides being a super hacker). I was a bit distracted the last day of the hackfest because of a small scare that ended up with me cancelling the talk I was going to give and rushing off to two Prague hospitals. I hope I didn’t worry everyone too much as it turned out to be a minor issue that was taken care of with some antibiotics. Tomeu was key in getting me where I needed to go.

A hundred things were going through my head as I had never had medical issues while being abroad – should I wait for my flight back home in three days, should I get emergency tickets back home, how much would it cost to go to a local hospital? Luckily my doctors office has web apps set up so I could securely be in communication with them while I was deciding what to do. I found out my insurance company would cover me though as it ended up Prague hospital care was a little more than what my copay would be. it all worked out in the end.

I promised the board to write-up a how-to on setting up a successful hackfest. I believe I will add a section on handling issues that may come up. We should have a policy of listing emergency information on the wiki of events we hold, including the closest places to seek care and approximate costs for travellers abroad. It is important to keep out guests safe and make them feel comfortable when they take time to help make GNOME better.

The hackfest is over and a lot of things got fixed but unfortunately Python3 support was broken.  I just committed a couple of patches to fix this.  Here are some things PyGObject developers should look out for. When I note to only use something in tests it means that there is a compat module that is only available to the tests. Actual code should either add the workaround to the top of their module or try not to have a distinction between things such as unicode and longs which no longer exist in Python 3.

• use range instead of xrange – loss of performance in Python 2 but Python 3 treats range similarly to python 2’s xrange
• use dict.items() instead of dict.iteritems() – same as the xrange issue
• callable does not exist in 3.x, use hasattr(obj, ‘__call__’) or

            if sys.version_info > (3, 0):
                def callable(obj):
                    return hasattr(obj, '__call__')
      

• using unicode in tests is tricky, you can’t use u” even in a versioned conditional as python3’s parser chokes on it. Do this in tests (and only i
  in tests):

            from compathelper import _unicode
            unicode_string = _unicode('this is a unicode string')
      

• exception caching changed in 2.7, instead of except Exception, e we now use except Exception as e. Do this to be compatible with older versions:

            except Exception:
                etype, e = sys.exc_info()[:2]
      

• Unbound methods with an im_func attribute no longer exits in 3.x. Unbound methods are now just functions so class.method in 3.x is equivalent to class.method.im_func in 2.x. If you have to go this low level do this:

            func = class.method
            if sys.version_info < (3,0):
                func = func.im_func
      

  Update: It was pointed out to me that this can be expressed as a simple getattr triple:

            getattr(class.method, 'im_func', class.method)
      

• all numbers are long in 3.x so 42L is invalid in 3.x. In tests (and only in tests) do this:

            from compathelper import _long
            l = _long(42)
      

  [read this post in: ar de es fr it ja ko pt ru zh-CN ]
   

It is all one big blur

We’ve been busy at the brmlab hacker space for a couple of days now hacking away at PyGObject Introspection.  A lot of bugs have been fixed and some of the biggest GEdit plugins now works quite well thanks to Ignacio.  Martin Pit has been working on porting system-config-printer which touches a lot of the Gtk+ API which he fixes as he runs into issues.  Right now he is working on making GVariants easier to use.  Nud has been hard at work tracking down reference counting issues which no one else wants to touch due to having to deal with two ref systems and a garbage collector.  Simon Schampijer is working on Gdk issues and is readying a patch to fix places where we don’t quite get constructors correct. Sebastian is porting gnome-dvb and Tomeu is unbreaking Cairo in PyGI after the Gtk+ guys decided to wrap Cairo structs in full boxed types.   Lazlo is adding annotations to Gtk+ and reviewing some of the annotations in Pavel’s gobject-introspection branch. I’ve been working on the new PyGI invoke branch which has been going really well but most likely won’t be ready to swap over until after the hackfest.

One of the greatest things about the hackfest is getting the PyGI developers familiar with debugging some of the harder areas of the library.  After this is over in a few more days everyone will go home with a greater knowledge of the different bits that make up the PyGI infrastructure.  This means we can start to accelerate porting and bug fixing of a few apps up to the release of GNOME 3 and then be in a good place to move everything over in the next cycle.

Thanks goes to Collabora for funding our beers which puts us to sleep so we can wake up refreshed the next morning.  The Collabora thank you dinner will be happening tomorrow night.   Also, thanks to the Foundation for giving us the infrastructure for organising such events.  I plan on writing something less formal than the current guidelines so other people can use as another reference to organizing similar events.  BTW, any hackers near Prague should check out the hackspace.  We need one of these in every city there is FOSS developers.

I got to a bit of a milestone today on the new PyGObject Introspection invoke code I have been working on. I can now run a handful of tests that marshal in’s out’s and returns. It is mostly just basic types right now but it works. Of significant importance is that I got the first torture test to run. We call a simple interface with a couple of in and out parameters 10000 times in a loop. Here is the output from the old implementation:

test_torture_profile (test_everything.TestTortureProfile) ...
        torture test 1 (10000 iterations): 0.240000 secs

and now from my new implementation:

test_torture_profile (test_cache.TestTortureProfile) ...
	torture test 1 (10000 iterations): 0.070000 secs

That is more than a 3x speedup for a simple case. Of course there is still a lot of work to do to handle more complex types and all of the edge cases but again, progress. I’m probably losing some speed gains due to moving to function calls instead of a big switch statement but one of the benefits of splitting everything up is when issues occur I know exactly where it is happening instead of having to scroll up the code to see if I am decoding or encoding and what type is causing the issues.

The hackfest is next week in Prague. Note to those going, because of the small amounts involved in your travel costs, I will be handling reimbursement in Euros or CZK. We will figure it all out when I get there.

I’ve been so busy I also forgot to thank Collabora who is sponsoring the hotel, a thank you dinner, beer and a coffee machine for the hackspace which is donating rooms for us to hack in. I’m looking forward to having a great time and getting some work done. Hopefully this blizzard that is hitting us tonight won’t effect my travel.

I’ve gotten pretty far with the new caching layer for PyGObject. Nothing compiles yet but the skeleton code is sitting on the invoke-rewrite branch. Right now I am working with the “in” marshallers and validators. I came to the conclusion that validation and marshalling are really intertwined so it is much clearer to do the validation during the marshalling step. So now since I am breaking things up into cachable functions, this hot mess:

UINT8 validation

        case GI_TYPE_TAG_UINT8:
            /* UINT8 types can be characters */
            if (PYGLIB_PyBytes_Check(object)) {
                if (PYGLIB_PyBytes_Size(object) != 1) {
                    PyErr_Format (PyExc_TypeError, "Must be a single character");
                    retval = 0;
                    break;
                }

                break;
            }
        case GI_TYPE_TAG_INT8:
        case GI_TYPE_TAG_INT16:
        case GI_TYPE_TAG_UINT16:
        case GI_TYPE_TAG_INT32:
        case GI_TYPE_TAG_UINT32:
        case GI_TYPE_TAG_INT64:
        case GI_TYPE_TAG_UINT64:
        case GI_TYPE_TAG_FLOAT:
        case GI_TYPE_TAG_DOUBLE:
        {
            PyObject *number, *lower, *upper;

            if (!PyNumber_Check (object)) {
                PyErr_Format (PyExc_TypeError, "Must be number, not %s",
                              object->ob_type->tp_name);
                retval = 0;
                break;
            }

            if (type_tag == GI_TYPE_TAG_FLOAT || type_tag == GI_TYPE_TAG_DOUBLE) {
                number = PyNumber_Float (object);
            } else {
                number = PYGLIB_PyNumber_Long (object);
            }

            _pygi_g_type_tag_py_bounds (type_tag, &lower, &upper);

            if (lower == NULL || upper == NULL || number == NULL) {
                retval = -1;
                goto check_number_release;
            }

            /* Check bounds */
            if (PyObject_RichCompareBool (lower, number, Py_GT)
                    || PyObject_RichCompareBool (upper, number, Py_LT)) {
                PyObject *lower_str;
                PyObject *upper_str;

                if (PyErr_Occurred()) {
                    retval = -1;
                    goto check_number_release;
                }

                lower_str = PyObject_Str (lower);
                upper_str = PyObject_Str (upper);
                if (lower_str == NULL || upper_str == NULL) {
                    retval = -1;
                    goto check_number_error_release;
                }

#if PY_VERSION_HEX < 0x03000000
                PyErr_Format (PyExc_ValueError, "Must range from %s to %s",
                              PyString_AS_STRING (lower_str),
                              PyString_AS_STRING (upper_str));
#else
                {
                    PyObject *lower_pybytes_obj = PyUnicode_AsUTF8String (lower_str);
                    if (!lower_pybytes_obj)
                        goto utf8_fail;

                    PyObject *upper_pybytes_obj = PyUnicode_AsUTF8String (upper_str);
                    if (!upper_pybytes_obj) {
                        Py_DECREF(lower_pybytes_obj);
                        goto utf8_fail;
                    }

                    PyErr_Format (PyExc_ValueError, "Must range from %s to %s",
                                  PyBytes_AsString (lower_pybytes_obj),
                                  PyBytes_AsString (upper_pybytes_obj));
                    Py_DECREF (lower_pybytes_obj);
                    Py_DECREF (upper_pybytes_obj);
                }
utf8_fail:
#endif
                retval = 0;

check_number_error_release:
                Py_XDECREF (lower_str);
                Py_XDECREF (upper_str);
            }

check_number_release:
            Py_XDECREF (number);
            Py_XDECREF (lower);
            Py_XDECREF (upper);
            break;
        }

UINT8 marshalling

        case GI_TYPE_TAG_UINT8:
            if (PYGLIB_PyBytes_Check(object)) {
                arg.v_long = (long)(PYGLIB_PyBytes_AsString(object)[0]);
                break;
            }

        case GI_TYPE_TAG_INT8:
        case GI_TYPE_TAG_INT16:
        case GI_TYPE_TAG_UINT16:
        case GI_TYPE_TAG_INT32:
        {
            PyObject *int_;

            int_ = PYGLIB_PyNumber_Long (object);
            if (int_ == NULL) {
                break;
            }

            arg.v_long = PYGLIB_PyLong_AsLong (int_);

            Py_DECREF (int_);

            break;
        }

becomes this easier to read function:

UINT8 cached marshalling and validation function

gboolean
_pygi_marshal_in_uint8 (PyGIState         *state,
                        PyGIFunctionCache *function_cache,
                        PyGIArgCache      *arg_cache,
                        PyObject          *py_arg,
                        GIArgument        *arg)
{
    long long_;

    if (PYGLIB_PyBytes_Check(py_arg)) { 

        if (PYGLIB_PyBytes_Size(py_arg) != 1) {
            PyErr_Format (PyExc_TypeError, "Must be a single character");
            return FALSE;
        }

        long_ = (long)(PYGLIB_PyBytes_AsString(py_arg)[0]);

    } else if (PyNumber_Check(py_arg)) {
        PyObject *py_long;
        py_long = PYGLIB_PyNumber_Long(py_arg);
        if (!py_long)
            return FALSE;

        long_ = PYGLIB_PyLong_AsLong(py_long);
        Py_DECREF(py_long);

        if (PyErr_Occured())
            return FALSE;
    } else {
        PyErr_Format (PyExc_TypeError, "Must be number or single byte string, not %s",
                      py_arg->ob_type->tp_name);
        return FALSE;
    }

    if (long_ < 0 || long_ > 255) {
        PyErr_Format (PyExc_ValueError, "%li not in range %i to %i", long_, 0, 255);
        return FALSE;
    }

    arg.v_long = long_;

    return TRUE;
}

Notice the if (long_ < 0 || long_ > 255) check. With the previous version we were actually converting the ranges to python objects and comparing them via the python RichCompare interface. Now since we have to decode the value anyway, we can just do a simple C comparison which is an order of magnitude faster. This doesn't even show some of the areas where we are decoding twice because of the split validation and marshalling routines. In any case ... progress ... though we won't know how much until I have implemented enough to get it running the test suite.

Other advantage come from reorganizing invoke including preliminary support for default values. I added a way for user_data to default to NULL if not passed in by the user. It is designed in such a way that any default value can be added to an argument cache and invoke should be able to substitute it. This means that once GI adds a way to specify default values we simply need to marshal them into the the corresponding cache and it should just work. Another enhancement that should be easy to add later is keyword argument support. Since we are normalizing the data in the cache, theoretically one should be able to pick a random argument and marshal it without having to process the argument list linearly. That however will be a later feature.