[plum] Structure Tutorial: Variable Member Types

This tutorial shows how to use the TypeMember and VariableTypeMember member definitions within a structure definition to control one member’s type based on the value of another structure member.

To demonstrate, examine the example below. The datatype member controls the data member’s type. Usage of TypeMember defines the structure member that controls the variable type. Usage of VariableTypeMember defines the member who’s type is being controlled by the mapping.

During unpacking of the structure, the the value of the unpacked datatype member determines the type of the data member per the defined mapping:

>>> from plum.int.little import UInt8, UInt16, UInt32
>>> from plum.structure import Structure, TypeMember, VariableTypeMember
>>>
>>> TYPE_MAP = {0: UInt8, 1: UInt16, 2: UInt32}
>>>
>>> class Struct1(Structure):
...     datatype: int = TypeMember(cls=UInt8, mapping=TYPE_MAP)
...     data: int = VariableTypeMember(type_member=datatype)
...
>>> struct1 = Struct1.unpack(b'\x01\x02\x00')
>>> struct1
Struct1(datatype=1, data=2)
>>> struct1.dump()
+--------+-----------------+-------+-------+---------+
| Offset | Access          | Value | Bytes | Type    |
+--------+-----------------+-------+-------+---------+
|        |                 |       |       | Struct1 |
| 0      | [0] (.datatype) | 1     | 01    | UInt8   |
| 1      | [1] (.data)     | 2     | 02 00 | UInt16  |
+--------+-----------------+-------+-------+---------+

Tip

Consider using an integer enumeration for the datatype member type. See the quick start link on the left for information on enumeration types.

Similarly packing operations (either through direct instantiation or with a dictionary of members), the value of the datatype member determines the format of the data member bytes:

>>> # datatype -> UInt8 (0)
>>> Struct1(datatype=0, data=2).pack()  # pack via direct instantiation
bytearray(b'\x00\x02')
>>>
>>> # datatype -> UInt8 (1)
>>> Struct1.pack({'datatype': 1, 'data': 2})  # pack via dictionary of members
bytearray(b'\x01\x02\x00')

The structure may be instantiated or packed without needing to specify the datatype member since the defined association allows it to be automatically determined when the data member is of a type within the mapping:

>>> struct1 = Struct1(data=UInt16(2))
>>> struct1.dump()
+--------+-----------------+-------+-------+---------+
| Offset | Access          | Value | Bytes | Type    |
+--------+-----------------+-------+-------+---------+
|        |                 |       |       | Struct1 |
| 0      | [0] (.datatype) | 1     | 01    | UInt8   |
| 1      | [1] (.data)     | 2     | 02 00 | UInt16  |
+--------+-----------------+-------+-------+---------+
>>> Struct1.pack({'data': UInt16(2)})
bytearray(b'\x01\x02\x00')

The association causes the datatype member to be updated when setting a new data value using the data attribute:

>>> struct1 = Struct1(data=UInt16(2))
>>> struct1
Struct1(datatype=1, data=UInt16(2))
>>>
>>> struct1.data = UInt8(3)
>>> struct1
Struct1(datatype=0, data=UInt8(3))

Ignoring Members

Both TypeMember and VariableTypeMember support marking the member as to be ignored during comparisons and may be used at the same time for that purpose:

>>> from plum.int.little import UInt8, UInt16, UInt32
>>> from plum.structure import Member, Structure, TypeMember, VariableTypeMember
>>>
>>> TYPE_MAP = {0: UInt8, 1: UInt16, 2: UInt32}
>>>
>>> class Struct3(Structure):
...     datatype: int = TypeMember(cls=UInt8, mapping=TYPE_MAP, ignore=True)
...     data: int = VariableTypeMember(type_member=datatype, ignore=True)
...     bookend: int = Member(cls=UInt8)
...
>>> struct3 = Struct3.unpack(b'\x01\x02\x00\x99')
>>> struct3.dump()
+--------+-----------------+-------+-------+---------+
| Offset | Access          | Value | Bytes | Type    |
+--------+-----------------+-------+-------+---------+
|        |                 |       |       | Struct3 |
| 0      | [0] (.datatype) | 1     | 01    | UInt8   |
| 1      | [1] (.data)     | 2     | 02 00 | UInt16  |
| 3      | [2] (.bookend)  | 153   | 99    | UInt8   |
+--------+-----------------+-------+-------+---------+
>>>
>>> struct3 == Struct3(datatype=0, data=0, bookend=0x99)
True

Defaulting Members

VariableTypeMember supports specifying a default that is used during instantiation or packing operations when a value is not provided:

>>> from plum.int.little import UInt8, UInt16, UInt32
>>> from plum.structure import Structure, TypeMember, VariableTypeMember
>>>
>>> TYPE_MAP = {0: UInt8, 1: UInt16, 2: UInt32}
>>>
>>> class Struct4(Structure):
...     datatype: int = TypeMember(cls=UInt8, mapping=TYPE_MAP)
...     data: int = VariableTypeMember(type_member=datatype, default=UInt16(2))
...
>>> Struct4().dump()
+--------+-----------------+-------+-------+---------+
| Offset | Access          | Value | Bytes | Type    |
+--------+-----------------+-------+-------+---------+
|        |                 |       |       | Struct4 |
| 0      | [0] (.datatype) | 1     | 01    | UInt8   |
| 1      | [1] (.data)     | 2     | 02 00 | UInt16  |
+--------+-----------------+-------+-------+---------+

Custom Variable Member Type

The Member definition class accepts factory functions (or properties) for both the cls and default arguments. Under the hood, the TypeMember and VariableTypeMember use these features to support variably typed members within a structure that use a simple mapping. The following example shows a more sophisticated application of factory functions that determine the type based the magnitude of the value.

>>> class Sample(Structure):
...
...     def determine_datatype(self):
...         return 1 if self.data >> 8 else 0
...
...     @property
...     def data_cls(self):
...         return UInt16 if self.datatype else UInt8
...
...     def set_data(self, value):
...         self[1] = value  # use index to avoid recursion
...         self.datatype = self.determine_datatype()
...
...     datatype: int = Member(cls=UInt8, default=determine_datatype)
...     data: int = Member(cls=data_cls, setter=set_data)

The data_cls property (which could have been a simple function by leaving @property decorator off) used as the cls of the data member controls the class used when packing and unpacking the data member. The data_cls factory function facilitates leaving the datatype member unspecified when instantiation or packing and having it determined based on the value magnitude in the data member. For example:

>>> # integer that fits in a UInt8
>>> Sample(data=0).dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 0     | 00    | UInt8  |
| 1      | [1] (.data)     | 0     | 00    | UInt8  |
+--------+-----------------+-------+-------+--------+
>>>
>>> # integer that requires a UInt16
>>> Sample(data=256).dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 1     | 01    | UInt8  |
| 1      | [1] (.data)     | 256   | 00 01 | UInt16 |
+--------+-----------------+-------+-------+--------+

The set_data function passed as the setter argument in the data member definition overrides the normal attribute set mechanism for the data member. In addition to storing the new value, set_data in this example also updates the datatype member to reflect the new data value.

>>> # 'datatype' starts out as UInt8
>>> sample = Sample(data=0)
>>> sample.dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 0     | 00    | UInt8  |
| 1      | [1] (.data)     | 0     | 00    | UInt8  |
+--------+-----------------+-------+-------+--------+
>>>
>>> # but changes based on setting 'data' attribute to a new value
>>> sample.data = 256
>>> sample.dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 1     | 01    | UInt8  |
| 1      | [1] (.data)     | 256   | 00 01 | UInt16 |
+--------+-----------------+-------+-------+--------+

Instead of providing a default factory function for the datatype member, consider providing a complete implementation for __init__ (rather than relying on the __init__ that would otherwise be generated). Besides being more explicit, it helps IDEs offer type ahead hints when instantiating the structure:

>>> class Sample(Structure):
...
...     def __init__(self, data, datatype=None):
...         if datatype is None:
...             datatype = 1 if data >> 8 else 0
...
...         list.extend(self, [datatype, data])
...
...     @property
...     def data_cls(self):
...         return UInt16 if self.datatype else UInt8
...
...     def set_data(self, value):
...         self[1] = value  # use index to avoid recursion
...         self.datatype = 1 if value >> 8 else 0
...
...     datatype: int = Member(cls=UInt8)  # no default
...     data: int = Member(cls=data_cls, setter=set_data)
...
>>> # and works the same
>>> sample = Sample(data=0)
>>> sample.dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 0     | 00    | UInt8  |
| 1      | [1] (.data)     | 0     | 00    | UInt8  |
+--------+-----------------+-------+-------+--------+
>>>
>>> sample.data = 256
>>> sample.dump()
+--------+-----------------+-------+-------+--------+
| Offset | Access          | Value | Bytes | Type   |
+--------+-----------------+-------+-------+--------+
|        |                 |       |       | Sample |
| 0      | [0] (.datatype) | 1     | 01    | UInt8  |
| 1      | [1] (.data)     | 256   | 00 01 | UInt16 |
+--------+-----------------+-------+-------+--------+