Serialization¶
Introduction¶
osBrain uses by default the pickle
module for serialization when
passing messages between agents. Serialization is, however, configurable.
osBrain can also use dill
, cloudpickle
, json
and raw
. Where raw
means, actually, no serialization (i.e.: the user is
expected to send only raw bytes over those sockets).
Warning
Using some serializers such as pickle
, dill
or cloudpickle
can be security risk. Those serializers allow
arbitrary code execution while deserializing data and therefore may wreck
or compromise your system. Only use those when communications between
agents are secured (i.e.: encrypted or in a local area network).
Note
Note that different serializers might have different limitations.
For example, 'json'
does not support serializing an object of bytes
type, while 'pickle'
does support it.
Defining the serializer¶
Specifying the serializer only makes sense in server
sockets, since clients
will automatically detect and set the type they need in order to communicate
accordingly with the server.
There are three ways in which the serializer can be specified:
- Global configuration.
- Specifying it at
per-agent
configuration. - Specifying it at
per-socket
configuration.
Global configuration¶
By setting the osbrain.config['SERIALIZER']
configuration variable, we
can change the default serializer. For example:
osbrain.config['SERIALIZER'] = 'json'
Note
It is also possible to change the global default serializer by
setting the OSBRAIN_DEFAULT_SERIALIZER
environment variable.
Per-agent configuration¶
Specifying the serializer at per-agent level will override the global configuration. This can be done as follows:
a1 = run_agent('a1', serializer='json')
Per-socket configuration¶
Finally, we can specify the serializer at per-socket level. This will override any other configuration (global/per-agent). For example:
a1 = run_agent('a1', serializer='json')
# Raw serialization will override json for this socket
addr1 = a1.bind('PUB', 'alias1', serializer='raw')
PUBSUB messaging pattern¶
For the PUBSUB pattern, there is a special character (b'x80'
as of now, even
though it could change at any time) that we use so as to let the agents know
what is the topic and what is the message itself. Note that the special
separator character is only required if there is a topic and the serialization
option is NOT set to raw
(read below for more information).
Considerations when using raw
serialization and PUBSUB pattern¶
Special care must be taken when working with raw
serialization and the PUBSUB
messaging pattern. Under those conditions, we decided to replicate the raw
ZeroMQ PUBSUB communication, in which the topic is sent along with the message
and is the handler the one that must take care of separating the topic from the
message it self.
Note that if we are using other type of serialization, it is safe to assume that what we are receiving only the original message, without any traces of the topic.