[plum] Tutorial: Arbitrary Formats

The format fmt specifier argument for the pack() and unpack() utility functions offer tremendous flexibility. This tutorial details the various possible constructs and their behavior.

Transform

When a transform is specified for fmt, packing uses it to produce bytes from the value provided:

>>> from plum.bigendian import uint8
>>> from plum.utilities import pack, unpack
>>>
>>> fmt = uint8
>>>
>>> buffer = pack(0, fmt)
>>> buffer
b'\x00'

Unpacking produces a Python object per the transform specified as fmt:

>>> unpack(fmt, buffer)
0

List

When a list of transforms are specified for fmt, packing uses them to produce bytes from the list (or any iterable) of values provided:

>>> fmt = [uint8, uint8, uint8]
>>> buffer = pack([0, 1, 2], fmt)
>>> buffer
b'\x00\x01\x02'

Unpacking produces a list of Python objects per the transforms specified:

>>> unpack(fmt, buffer)
[0, 1, 2]

Tuple

Providing a tuple of transforms for the fmt works exactly like providing a list except unpacking produces a tuple of Python objects instead of a list of Python objects:

>>> fmt = (uint8, uint8, uint8)
>>> buffer = pack([0, 1, 2], fmt)
>>> buffer
b'\x00\x01\x02'
>>> unpack(fmt, buffer)
(0, 1, 2)

Dict

Providing a dictionary of transforms works similarly except when packing operations expect a dictionary of values and unpacking produces a dictionary of Python objects:

>>> fmt = {"a": uint8, "b": uint8}
>>> buffer = pack({"a": 0, "b": 1}, fmt)
>>> buffer
b'\x00\x01'
>>> unpack(fmt, buffer)
{'a': 0, 'b': 1}

Nesting

pack() and unpack() accept arbitrarily nested dictionaries, lists, and tuples of transforms for fmt. For example:

>>> fmt = [(uint8, uint8), {"a": uint8, "b": uint8}]
>>> buffer = pack([(0, 1), {"a": 2, "b": 3}], fmt)
>>> buffer
b'\x00\x01\x02\x03'
>>> unpack(fmt, buffer)
[(0, 1), {'a': 2, 'b': 3}]

None

The pack() utility function supports None as the fmt for applications where the value is a data store with transform properties (e.g. BitFields, Structure):

>>> from plum.structure import Structure, member
>>>
>>> class MyStruct(Structure):
...     a: int = member(fmt=uint8)
...     b: int = member(fmt=uint8)
>>>
>>> # value is a data store with transform properties
>>> pack(MyStruct(a=1, b=2), fmt=None)
b'\x01\x02'
>>>
>>> # fmt defaults to None, no need to specify it!
>>> pack(MyStruct(a=1, b=2))
b'\x01\x02'

If the value is already bytes, pack() passes them through:

>>> pack(b'\x00\x01', fmt=None)
b'\x00\x01'

Similarly, pack() supports None within any dictionary, list, or tuple of transforms used as the fmt:

>>> fmt = [uint8, None]
>>> pack([0, MyStruct(a=1, b=2)], fmt)
b'\x00\x01\x02'
>>> pack([0, b'\x01\x02'], fmt)
b'\x00\x01\x02'

In places where None was specified, pack() also accepts a (value, fmt) tuple pair:

>>> pack([(0, uint8), (1, uint8)], fmt=[None, None])
b'\x00\x01'

Where ever fmt is None, pack() accepts dictionaries, lists, and tuples of (value, fmt) pairs for the value allowing complex structures to be accepted:

>>> # explicit fmt=None
>>> pack([(0, uint8), {"a": (1, uint8), "b": (2, uint8)}], fmt=None)
b'\x00\x01\x02'
>>>
>>> # implicit fmt=None
>>> pack([(0, uint8), {"a": (1, uint8), "b": (2, uint8)}])
b'\x00\x01\x02'
>>>
>>> # mix
>>> pack([0, {"a": (1, uint8), "b": (2, uint8)}], fmt=[uint8, None])
b'\x00\x01\x02'

When fmt is None, the unpack() utility function returns the remaining available bytes:

>>> unpack(fmt=None, buffer=b'\x00\x01')
b'\x00\x01'
>>> unpack(fmt=[uint8, None], buffer=b'\x00\x01\x02')
[0, b'\x01\x02']