Serialization

Introduction

osBrain uses by default the pickle module for serialization when passing messages between agents. Serialization is, however, configurable. osBrain can also use dill, cloudpickle, json and raw. Where raw means, actually, no serialization (i.e.: the user is expected to send only raw bytes over those sockets).

Warning

Using some serializers such as pickle, dill or cloudpickle can be security risk. Those serializers allow arbitrary code execution while deserializing data and therefore may wreck or compromise your system. Only use those when communications between agents are secured (i.e.: encrypted or in a local area network).

Note

Note that different serializers might have different limitations. For example, 'json' does not support serializing an object of bytes type, while 'pickle' does support it.

Defining the serializer

Specifying the serializer only makes sense in server sockets, since clients will automatically detect and set the type they need in order to communicate accordingly with the server.

There are three ways in which the serializer can be specified:

  • Global configuration.
  • Specifying it at per-agent configuration.
  • Specifying it at per-socket configuration.

Global configuration

By setting the osbrain.config['SERIALIZER'] configuration variable, we can change the default serializer. For example:

osbrain.config['SERIALIZER'] = 'json'

Note

It is also possible to change the global default serializer by setting the OSBRAIN_DEFAULT_SERIALIZER environment variable.

Per-agent configuration

Specifying the serializer at per-agent level will override the global configuration. This can be done as follows:

a1 = run_agent('a1', serializer='json')

Per-socket configuration

Finally, we can specify the serializer at per-socket level. This will override any other configuration (global/per-agent). For example:

a1 = run_agent('a1', serializer='json')
# Raw serialization will override json for this socket
addr1 = a1.bind('PUB', 'alias1', serializer='raw')

PUBSUB messaging pattern

For the PUBSUB pattern, there is a special character (b'\x80' as of now, even though it could change at any time) that we use so as to let the agents know what is the topic and what is the message itself. Note that the special separator character is only required if there is a topic and the serialization option is NOT set to raw (read below for more information).

Considerations when using raw serialization and PUBSUB pattern

Special care must be taken when working with raw serialization and the PUBSUB messaging pattern. Under those conditions, we decided to replicate the raw ZeroMQ PUBSUB communication, in which the topic is sent along with the message and is the handler the one that must take care of separating the topic from the message it self.

Note that if we are using other type of serialization, it is safe to assume that what we are receiving only the original message, without any traces of the topic.