[plum] Structure Tutorial: Sized Objects¶
This tutorial shows how to use the SizeMember and VariableSizeMember()
member definitions within a structure definition to associate a size member and another
variably sized member to one another. During unpack operations, the association
causes the the unpacked size member to control the number of bytes that are available
during the unpacking of the associated member. During dictionary pack operations
or during structure instantiations, the association allows the size member to be
left unspecified and the value of the size member derived from the other member.
To demonstrate, examine the example below. The example uses the Array
type since it acts greedy (it varies in size and consumes all bytes
given to it). If left to unpack as normal, the array member consumes every byte
including the byte intended for the bookmark. The use of SizeMember
and VariableSizeMember() member definitions associate the array member
with the size member. This causes the unpacking operation to only allow the
array member to consume the number of bytes specified by the size member.
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct1(Structure):
... size: int = SizeMember(cls=UInt8)
... array: list = VariableSizeMember(size_member=size, cls=Array)
... bookend: int = Member(cls=UInt8)
...
>>> Struct1.unpack(b'\x02\x01\x02\x99').dump()
+--------+----------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+-------+-------+---------+
| | | | | Struct1 |
| 0 | [0] (.size) | 2 | 02 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 1 | 01 | UInt8 |
| 2 | [1] | 2 | 02 | UInt8 |
| 3 | [2] (.bookend) | 153 | 99 | UInt8 |
+--------+----------------+-------+-------+---------+
This association also the size to be left unspecified
when instantiating the structure with member values:
>>> Struct1(array=[1, 2], bookend=0x99).dump()
+--------+----------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+-------+-------+---------+
| | | | | Struct1 |
| 0 | [0] (.size) | 2 | 02 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 1 | 01 | UInt8 |
| 2 | [1] | 2 | 02 | UInt8 |
| 3 | [2] (.bookend) | 153 | 99 | UInt8 |
+--------+----------------+-------+-------+---------+
>>>
>>> Struct1.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x02\x01\x02\x99')
Similarly, when packing a dictionary of member values using a structure type,
the size member may be left unspecified:
>>> Struct1.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x02\x01\x02\x99')
The association causes the size member to be updated when setting
a new array value using the array attribute:
>>> struct1 = Struct1(array=[1, 2], bookend=123)
>>> struct1
Struct1(size=2, array=[1, 2], bookend=123)
>>>
>>> struct1.array = [1, 2, 3, 4, 5]
>>> struct1
Struct1(size=5, array=[1, 2, 3, 4, 5], bookend=123)
Using Ratios¶
SizeMember supports accepting a ratio parameter to facilitate
converting the size member units to a byte count when it is not a 1:1 ratio.
In the example below, a ratio of 2 means that for every size increment of
1, the array holds 2 extra bytes:
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct2(Structure):
... size: int = SizeMember(cls=UInt8, ratio=2)
... array: list = VariableSizeMember(size_member=size, cls=Array)
... bookend: int = Member(cls=UInt8)
...
>>> Struct2.unpack(b'\x01\x01\x02\x99').dump()
+--------+----------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+-------+-------+---------+
| | | | | Struct2 |
| 0 | [0] (.size) | 1 | 01 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 1 | 01 | UInt8 |
| 2 | [1] | 2 | 02 | UInt8 |
| 3 | [2] (.bookend) | 153 | 99 | UInt8 |
+--------+----------------+-------+-------+---------+
>>>
>>> Struct2(array=[1, 2], bookend=0x99).pack()
bytearray(b'\x01\x01\x02\x99')
>>>
>>> Struct2.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x01\x01\x02\x99')
Ignoring Members¶
SizeMember and VariableSizeMember() member definitions
support marking the members to be ignored during comparisons and may
be used at the same time for that purpose:
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>>
>>> class Struct4(Structure):
... size: int = SizeMember(cls=UInt8, ignore=True)
... array: list = VariableSizeMember(size_member=size, cls=Array, ignore=True)
... bookend: int = Member(cls=UInt8)
...
>>> struct4 = Struct4.unpack(b'\x02\x01\x02\x99')
>>> struct4.dump()
+--------+----------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+-------+-------+---------+
| | | | | Struct4 |
| 0 | [0] (.size) | 2 | 02 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 1 | 01 | UInt8 |
| 2 | [1] | 2 | 02 | UInt8 |
| 3 | [2] (.bookend) | 153 | 99 | UInt8 |
+--------+----------------+-------+-------+---------+
>>>
>>> struct4 == Struct4(array=[], bookend=0x99)
True
Defaulting Sized Member¶
Specifying a default value for the array in the member definition allows the structure to be instantiated or packed without a value provided for the array:
>>> from plum import pack
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>>
>>> class Struct5(Structure):
... size: int = SizeMember(cls=UInt8)
... array: list = VariableSizeMember(size_member=size, cls=Array, default=(0, 1))
... bookend: int = Member(cls=UInt8)
...
>>> Struct5(bookend=0x99).dump()
+--------+----------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+----------------+-------+-------+---------+
| | | | | Struct5 |
| 0 | [0] (.size) | 2 | 02 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 0 | 00 | UInt8 |
| 2 | [1] | 1 | 01 | UInt8 |
| 3 | [2] (.bookend) | 153 | 99 | UInt8 |
+--------+----------------+-------+-------+---------+
>>>
>>> pack(Struct5, {'bookend': 0x99})
bytearray(b'\x02\x00\x01\x99')
Accomodating Fixed Offset¶
The offset argument of the SizeMember compensates for
the size member reflecting both the size of the variable size member
and overhead bytes such as the size member itself. For example,
the size member in the following structure reflects the total size
of the structure including the size member:
>>> from plum import pack
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, SizeMember, Structure, VariableSizeMember
>>>
>>> class Struct6(Structure):
... size: int = SizeMember(cls=UInt8, offset=1)
... array: list = VariableSizeMember(size_member=size, cls=Array)
...
>>> Struct6(array=[0, 1]).dump()
+--------+--------------+-------+-------+---------+
| Offset | Access | Value | Bytes | Type |
+--------+--------------+-------+-------+---------+
| | | | | Struct6 |
| 0 | [0] (.size) | 3 | 03 | UInt8 |
| | [1] (.array) | | | Array |
| 1 | [0] | 0 | 00 | UInt8 |
| 2 | [1] | 1 | 01 | UInt8 |
+--------+--------------+-------+-------+---------+