[plum] Structure Tutorial: Sized Arrays

This tutorial shows how to use the DimsMember and VariableDimsMember() member definitions within a structure definition to associate a dimensions member and an array member to one another. During unpack operations, the association causes the the unpacked dimension member to control the unpacking of the array member. During dictionary pack operations or during structure instantiations, the association allows the dimension member or the array member to be left unspecified and the value of the missing member derived from the other member.

One-Dimensional Arrays

To demonstrate, lets start by examining the example below. The example shows the simplest use case, an array with a single dimension. Since its a single dimension, lets choose the array dimension member name to be count with the type as UInt8. After the type annotation, use the DimsMember class to define a member to hold the dimensions giving it a scalar integer Plum type as the cls.

For the associated array member, use the VariableDimsMember class to associate it to the dims member and specify the array base class (an undimensioned Array subclass that defines the array’s item type).

The bookend member exists in the example to show that when the structure is unpacked, the dimensions count member restricts the array and leaves the last byte for the bookend:

>>> from plum import pack
>>> from plum.array import Array
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, Structure, DimsMember, VariableDimsMember
>>>
>>> class SampleArray(Array, item_cls=UInt8):
...     """Sample undimensioned array."""
...
>>> class Struct1(Structure):
...     count: int = DimsMember(cls=UInt8)
...     array: list = VariableDimsMember(dims_member=count, cls=SampleArray)
...     bookend: int = Member(cls=UInt8)
...
>>> Struct1.unpack(b'\x02\x01\x02\x99').dump()
+--------+----------------+-------+-------+-------------+
| Offset | Access         | Value | Bytes | Type        |
+--------+----------------+-------+-------+-------------+
|        |                |       |       | Struct1     |
| 0      | [0] (.count)   | 2     | 02    | UInt8       |
|        | [1] (.array)   |       |       | SampleArray |
| 1      |   [0]          | 1     | 01    | UInt8       |
| 2      |   [1]          | 2     | 02    | UInt8       |
| 3      | [2] (.bookend) | 153   | 99    | UInt8       |
+--------+----------------+-------+-------+-------------+

The association also allows the size member to be left unspecified when instantiating the structure. In that case, the size member takes on the length of the array:

>>> Struct1(array=[1, 2], bookend=0x99).dump()
+--------+----------------+-------+-------+-------------+
| Offset | Access         | Value | Bytes | Type        |
+--------+----------------+-------+-------+-------------+
|        |                |       |       | Struct1     |
| 0      | [0] (.count)   | 2     | 02    | UInt8       |
|        | [1] (.array)   |       |       | SampleArray |
| 1      |   [0]          | 1     | 01    | UInt8       |
| 2      |   [1]          | 2     | 02    | UInt8       |
| 3      | [2] (.bookend) | 153   | 99    | UInt8       |
+--------+----------------+-------+-------+-------------+

Similarly, when packing a dictionary of member values using a structure type, the count member may be left unspecified:

>>> Struct1.pack({'array': [1, 2], 'bookend': 0x99})
bytearray(b'\x02\x01\x02\x99')

The association causes the count member to be updated when setting a new array value using the array attribute:

>>> struct1 = Struct1(array=[1, 2], bookend=123)
>>> struct1
Struct1(count=2, array=[1, 2], bookend=123)
>>>
>>> struct1.array = [1, 2, 3, 4, 5]
>>> struct1
Struct1(count=5, array=[1, 2, 3, 4, 5], bookend=123)

Multi-Dimensional Arrays

Implementing multiple dimensions only adds slightly more complexity. Instead of supplying an scalar Plum integer type for the dimensions member, create a single dimension array subclass to hold the dimensions (e.g. for a two dimensional array, specify dims=(2,), for three dimensions dims=(3,), etc). Everything else remains the same:

>>> class Dims(Array, dims=(2,)):
...     pass
...
>>> class Struct2(Structure):
...     dims: list = DimsMember(cls=Dims)
...     array: list = VariableDimsMember(dims_member=dims, cls=SampleArray)
...
>>> Struct2(array=[[1, 2], [3, 4]]).dump()
+--------+--------------+-------+-------+-------------+
| Offset | Access       | Value | Bytes | Type        |
+--------+--------------+-------+-------+-------------+
|        |              |       |       | Struct2     |
|        | [0] (.dims)  |       |       | Dims        |
| 0      |   [0]        | 2     | 02    | UInt8       |
| 1      |   [1]        | 2     | 02    | UInt8       |
|        | [1] (.array) |       |       | SampleArray |
|        |   [0]        |       |       |             |
| 2      |     [0]      | 1     | 01    | UInt8       |
| 3      |     [1]      | 2     | 02    | UInt8       |
|        |   [1]        |       |       |             |
| 4      |     [0]      | 3     | 03    | UInt8       |
| 5      |     [1]      | 4     | 04    | UInt8       |
+--------+--------------+-------+-------+-------------+

Ignoring Members

Both DimsMember and VariableDimsMember support marking the member to be ignored during comparisons:

>>> class Struct1Ignore(Structure):
...     count: int = DimsMember(cls=UInt8, ignore=True)
...     array: list = VariableDimsMember(dims_member=count, cls=SampleArray, ignore=True)
...     bookend: int = Member(cls=UInt8)
...
>>> struct1 = Struct1Ignore.unpack(b'\x02\x01\x02\x99')
>>> struct1.dump()
+--------+----------------+-------+-------+---------------+
| Offset | Access         | Value | Bytes | Type          |
+--------+----------------+-------+-------+---------------+
|        |                |       |       | Struct1Ignore |
| 0      | [0] (.count)   | 2     | 02    | UInt8         |
|        | [1] (.array)   |       |       | SampleArray   |
| 1      |   [0]          | 1     | 01    | UInt8         |
| 2      |   [1]          | 2     | 02    | UInt8         |
| 3      | [2] (.bookend) | 153   | 99    | UInt8         |
+--------+----------------+-------+-------+---------------+
>>>
>>> struct1 == Struct1Ignore(array=[], bookend=0x99)
True

Defaulting Array Member

Specifying a default value for the array in its member definition allows the structure to be instantiated or packed without a value for the array member:

>>> class Struct1(Structure):
...     count: int = DimsMember(cls=UInt8)
...     array: list = VariableDimsMember(dims_member=count, cls=SampleArray, default=(0, 1))
...     bookend: int = Member(cls=UInt8)
...
>>> Struct1(bookend=0x99).dump()
+--------+----------------+-------+-------+-------------+
| Offset | Access         | Value | Bytes | Type        |
+--------+----------------+-------+-------+-------------+
|        |                |       |       | Struct1     |
| 0      | [0] (.count)   | 2     | 02    | UInt8       |
|        | [1] (.array)   |       |       | SampleArray |
| 1      |   [0]          | 0     | 00    | UInt8       |
| 2      |   [1]          | 1     | 01    | UInt8       |
| 3      | [2] (.bookend) | 153   | 99    | UInt8       |
+--------+----------------+-------+-------+-------------+
>>>
>>> pack(Struct1, {'bookend': 0x99})
bytearray(b'\x02\x00\x01\x99')