The data structures used to represent information and program state in memory are rarely conducive to the storage and transmission of that information and state. The process of translating those data structures into a format better suited to storage is known as serialisation or marshalling.

Google’s Protobuf is a mechanism for serialising and deserialising structured data. Pyrobuf is a alternative, lightning-fast, implementation of Protobuf written in Cython that is 2-4 times faster than Protobuf when using its C++ backend and 20-40 times faster than its pure Python implementation.

In this talk we’ll briefly review Protobuf and compare it to other well known serialisation mechanisms such as XML and JSON. We’ll then take a look at Pyrobuf and the basic strategies it uses to achieve significantly better performance.

Pyrobuf’s main claim to fame is its speed. This makes fixing bugs and adding new features a real challenge as any degradation in performance will not be well-received by users who’ve come to expect and rely on its speed. Therefore, for the remainder of the talk we’ll discuss tricks and tips, some more legitimate than others, for maintaining speed as new features are added.