[plum] Structure Tutorial: Sized Objects

This tutorial shows how to use the SizeMember and VariableSizeMember() member definitions within a structure definition to associate a size member and another variably sized member to one another. During unpack operations, the association causes the the unpacked size member to control the number of bytes that are available during the unpacking of the associated member. During dictionary pack operations or during structure instantiations, the association allows the size member to be left unspecified and the value of the size member derived from the other member.

To demonstrate, examine the example below. The example uses the Array type since it acts greedy (it varies in size and consumes all bytes given to it). If left to unpack as normal, the array member consumes every byte including the byte intended for the bookmark. The use of SizeMember and VariableSizeMember() member definitions associate the array member with the size member. This causes the unpacking operation to only allow the array member to consume the number of bytes specified by the size member.

>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct1(Structure):
...     size: int = SizeMember(cls=UInt8)
...     array: list = VariableSizeMember(size_member=size, cls=Array)
...     bookend: int = Member(cls=UInt8)
...
>>> Struct1.unpack(b'\x02\x01\x02\x99').dump()
+--------+----------------+-------+-------+---------+
| Offset | Access         | Value | Bytes | Type    |
+--------+----------------+-------+-------+---------+
|        |                |       |       | Struct1 |
| 0      | [0] (.size)    | 2     | 02    | UInt8   |
|        | [1] (.array)   |       |       | Array   |
| 1      |   [0]          | 1     | 01    | UInt8   |
| 2      |   [1]          | 2     | 02    | UInt8   |
| 3      | [2] (.bookend) | 153   | 99    | UInt8   |
+--------+----------------+-------+-------+---------+

This association also the size to be left unspecified when instantiating the structure with member values:

>>> Struct1(array=[1, 2], bookend=0x99).dump()
+--------+----------------+-------+-------+---------+
| Offset | Access         | Value | Bytes | Type    |
+--------+----------------+-------+-------+---------+
|        |                |       |       | Struct1 |
| 0      | [0] (.size)    | 2     | 02    | UInt8   |
|        | [1] (.array)   |       |       | Array   |
| 1      |   [0]          | 1     | 01    | UInt8   |
| 2      |   [1]          | 2     | 02    | UInt8   |
| 3      | [2] (.bookend) | 153   | 99    | UInt8   |
+--------+----------------+-------+-------+---------+
>>>
>>> Struct1.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x02\x01\x02\x99')

Similarly, when packing a dictionary of member values using a structure type, the size member may be left unspecified:

>>> Struct1.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x02\x01\x02\x99')

The association causes the size member to be updated when setting a new array value using the array attribute:

>>> struct1 = Struct1(array=[1, 2], bookend=123)
>>> struct1
Struct1(size=2, array=[1, 2], bookend=123)
>>>
>>> struct1.array = [1, 2, 3, 4, 5]
>>> struct1
Struct1(size=5, array=[1, 2, 3, 4, 5], bookend=123)

Using Ratios

SizeMember supports accepting a ratio parameter to facilitate converting the size member units to a byte count when it is not a 1:1 ratio. In the example below, a ratio of 2 means that for every size increment of 1, the array holds 2 extra bytes:

>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct2(Structure):
...     size: int = SizeMember(cls=UInt8, ratio=2)
...     array: list = VariableSizeMember(size_member=size, cls=Array)
...     bookend: int = Member(cls=UInt8)
...
>>> Struct2.unpack(b'\x01\x01\x02\x99').dump()
+--------+----------------+-------+-------+---------+
| Offset | Access         | Value | Bytes | Type    |
+--------+----------------+-------+-------+---------+
|        |                |       |       | Struct2 |
| 0      | [0] (.size)    | 1     | 01    | UInt8   |
|        | [1] (.array)   |       |       | Array   |
| 1      |   [0]          | 1     | 01    | UInt8   |
| 2      |   [1]          | 2     | 02    | UInt8   |
| 3      | [2] (.bookend) | 153   | 99    | UInt8   |
+--------+----------------+-------+-------+---------+
>>>
>>> Struct2(array=[1, 2], bookend=0x99).pack()
bytearray(b'\x01\x01\x02\x99')
>>>
>>> Struct2.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x01\x01\x02\x99')

Ignoring Members

SizeMember and VariableSizeMember() member definitions support marking the members to be ignored during comparisons and may be used at the same time for that purpose:

>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct4(Structure):
...     size: int = SizeMember(cls=UInt8, ignore=True)
...     array: list = VariableSizeMember(size_member=size, cls=Array, ignore=True)
...     bookend: int = Member(cls=UInt8)
...
>>> struct4 = Struct4.unpack(b'\x02\x01\x02\x99')
>>> struct4.dump()
+--------+----------------+-------+-------+---------+
| Offset | Access         | Value | Bytes | Type    |
+--------+----------------+-------+-------+---------+
|        |                |       |       | Struct4 |
| 0      | [0] (.size)    | 2     | 02    | UInt8   |
|        | [1] (.array)   |       |       | Array   |
| 1      |   [0]          | 1     | 01    | UInt8   |
| 2      |   [1]          | 2     | 02    | UInt8   |
| 3      | [2] (.bookend) | 153   | 99    | UInt8   |
+--------+----------------+-------+-------+---------+
>>>
>>> struct4 == Struct4(array=[], bookend=0x99)
True

Defaulting Sized Member

Specifying a default value for the array in the member definition allows the structure to be instantiated or packed without a value provided for the array:

>>> from plum import pack
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>>
>>> class Struct5(Structure):
...     size: int = SizeMember(cls=UInt8)
...     array: list = VariableSizeMember(size_member=size, cls=Array, default=(0, 1))
...     bookend: int = Member(cls=UInt8)
...
>>> Struct5(bookend=0x99).dump()
+--------+----------------+-------+-------+---------+
| Offset | Access         | Value | Bytes | Type    |
+--------+----------------+-------+-------+---------+
|        |                |       |       | Struct5 |
| 0      | [0] (.size)    | 2     | 02    | UInt8   |
|        | [1] (.array)   |       |       | Array   |
| 1      |   [0]          | 0     | 00    | UInt8   |
| 2      |   [1]          | 1     | 01    | UInt8   |
| 3      | [2] (.bookend) | 153   | 99    | UInt8   |
+--------+----------------+-------+-------+---------+
>>>
>>> pack(Struct5, {'bookend': 0x99})
bytearray(b'\x02\x00\x01\x99')

Accomodating Fixed Offset

The offset argument of the SizeMember compensates for the size member reflecting both the size of the variable size member and overhead bytes such as the size member itself. For example, the size member in the following structure reflects the total size of the structure including the size member:

>>> from plum import pack
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>>
>>> class Struct6(Structure):
...     size: int = SizeMember(cls=UInt8, offset=1)
...     array: list = VariableSizeMember(size_member=size, cls=Array)
...
>>> Struct6(array=[0, 1]).dump()
+--------+--------------+-------+-------+---------+
| Offset | Access       | Value | Bytes | Type    |
+--------+--------------+-------+-------+---------+
|        |              |       |       | Struct6 |
| 0      | [0] (.size)  | 3     | 03    | UInt8   |
|        | [1] (.array) |       |       | Array   |
| 1      |   [0]        | 0     | 00    | UInt8   |
| 2      |   [1]        | 1     | 01    | UInt8   |
+--------+--------------+-------+-------+---------+