Array to bytes and bytes to array transform.

[plum.array] Module Reference

The plum.array module provides the ArrayX transform which converts lists of uniformly typed items into bytes and bytes into lists of uniformly typed items. This reference page demonstrates creating and using an ArrayX transform as well as provides API details.

The examples shown on this page require the following setup:

>>> from plum.array import ArrayX
>>> from plum.bigendian import uint8, uint16
>>> from plum.utilities import pack_and_dump, unpack

Dimensioned Array

The ArrayX transform accepts the following arguments:

fmt:format of each array element
dims:length of each array dimension
name:transform name (for representations including dump format column)

The ArrayX transform fmt argument accepts a transform that describes the conversion between bytes and the array’s members.

The ArrayX transform dims argument accepts a tuple of integers that describe the length of each dimension of the array list.

>>> array_2x2 = ArrayX(fmt=uint16, dims=(2, 2))
>>> array_2x2
<transform 'List[List[int]]'>

When packing, the transform accepts any iterable (list, tuple, etc.) and ensures the iterable has the proper dimensions (and raises a PackError if not):

>>> array_2x2.pack([[1, 2], [3, 4]])
b'\x00\x01\x00\x02\x00\x03\x00\x04'
>>>
>>> array_2x2.pack([[1, 2], [3]])
Traceback (most recent call last):
    ...
plum.exceptions.PackError:
<BLANKLINE>
+--------+--------+-----------+-------+-----------------+
| Offset | Access | Value     | Bytes | Format          |
+--------+--------+-----------+-------+-----------------+
|        |        |           |       | List[List[int]] |
|        | [0]    |           |       |                 |
| 0      |   [0]  | 1         | 00 01 | uint16          |
| 2      |   [1]  | 2         | 00 02 | uint16          |
|        | [1]    |           |       |                 |
| 4      |   [0]  | 3         | 00 03 | uint16          |
|        |   [1]  | <missing> |       |                 |
+--------+--------+-----------+-------+-----------------+
<BLANKLINE>
TypeError occurred during pack operation:
<BLANKLINE>
invalid value, expected iterable of 2 length, got iterable of length 1

When unpacking, the transform produces a list of items from the unpacked bytes and uses the dims to control the size of the list for each dimension. If too few or too many bytes are provided, the array transform raises an UnpackError.

>>> buffer = b'\x00\x01\x00\x02\x00\x03\x00\x04'
>>>
>>> array_2x2.unpack(buffer)
[[1, 2], [3, 4]]
>>>
>>> array_2x2.unpack(buffer[:-1])
Traceback (most recent call last):
    ...
plum.exceptions.UnpackError:
<BLANKLINE>
+--------+--------+----------------------+-------+-----------------+
| Offset | Access | Value                | Bytes | Format          |
+--------+--------+----------------------+-------+-----------------+
|        |        |                      |       | List[List[int]] |
|        | [0]    |                      |       |                 |
| 0      |   [0]  | 1                    | 00 01 | uint16          |
| 2      |   [1]  | 2                    | 00 02 | uint16          |
|        | [1]    |                      |       |                 |
| 4      |   [0]  | 3                    | 00 03 | uint16          |
| 6      |   [1]  | <insufficient bytes> | 00    | uint16          |
+--------+--------+----------------------+-------+-----------------+
<BLANKLINE>
InsufficientMemoryError occurred during unpack operation:
<BLANKLINE>
1 too few bytes to unpack uint16, 2 needed, only 1 available

Greedy Array

When the dims argument is left to default to (None, ), the resulting transform becomes “greedy”. When packing, for any dimension specified as None, the transform accepts any number of items from the iterable (e.g. list, tuple, etc.) and packs each into a bytes buffer per the specified fmt.

>>> greedy_array = ArrayX(fmt=uint8)
>>> greedy_array
<transform 'List[int]'>
>>>
>>> buffer, dump = greedy_array.pack_and_dump([1, 2, 3])
>>> buffer
b'\x01\x02\x03'
>>> print(dump)
+--------+--------+-------+-------+-----------+
| Offset | Access | Value | Bytes | Format    |
+--------+--------+-------+-------+-----------+
|        |        |       |       | List[int] |
| 0      | [0]    | 1     | 01    | uint8     |
| 1      | [1]    | 2     | 02    | uint8     |
| 2      | [2]    | 3     | 03    | uint8     |
+--------+--------+-------+-------+-----------+

When unpacking, the transform consumes all remaining bytes in the buffer and produces a list of as many unpacked items as possible:

>>> array, dump = greedy_array.unpack_and_dump(b'\x01\x02\x03')
>>> array
[1, 2, 3]
>>> print(dump)
+--------+--------+-------+-------+-----------+
| Offset | Access | Value | Bytes | Format    |
+--------+--------+-------+-------+-----------+
|        |        |       |       | List[int] |
| 0      | [0]    | 1     | 01    | uint8     |
| 1      | [1]    | 2     | 02    | uint8     |
| 2      | [2]    | 3     | 03    | uint8     |
+--------+--------+-------+-------+-----------+

Note

The ArrayX transform accepts None for any of the dimensions in multi-dimensional applications, but limitations and special behaviors exist. For packing, for any dimension specified as None, the transform accepts any number of items for that particular dimension and no consistency checks are enforced. For unpacking, the transform only allows None for the outermost (first) dimension, otherwise a :class:UnpackError results.

Sized Arrays

The ArrayX transform constructor accepts an IntX transform instance. When unpacking, the transform first unpacks the dimension of the array from the byte sequence, then unpacks the array data accordingly. When packing, the transform includes the dimensions of the array data in the generated byte sequence. For example:

>>> single_dim_array = ArrayX(fmt=uint8, dims=uint16)
>>> bindata, dump = pack_and_dump([1, 2], fmt=single_dim_array)
>>> dump()
+--------+--------+-------+-------+-----------+
| Offset | Access | Value | Bytes | Format    |
+--------+--------+-------+-------+-----------+
|        |        |       |       | List[int] |
| 0      | len()  | 2     | 00 02 | uint16    |
| 2      | [0]    | 1     | 01    | uint8     |
| 3      | [1]    | 2     | 02    | uint8     |
+--------+--------+-------+-------+-----------+
>>>
>>> unpack(single_dim_array, bindata)
[1, 2]

Similarly, the ArrayX transform constructor also accepts an ArrayX transform instance for multiple dimensions:

>>> multi_dim_array = ArrayX(fmt=uint8, dims=ArrayX(fmt=uint8, dims=[2]))
>>> bindata, dump = pack_and_dump([[1, 2, 3], [4, 5, 6]], fmt=multi_dim_array)
>>> dump()
+--------+----------+-------+-------+-----------------+
| Offset | Access   | Value | Bytes | Format          |
+--------+----------+-------+-------+-----------------+
|        |          |       |       | List[List[int]] |
|        | --dims-- |       |       | List[int]       |
| 0      |   [0]    | 2     | 02    | uint8           |
| 1      |   [1]    | 3     | 03    | uint8           |
|        | [0]      |       |       |                 |
| 2      |   [0]    | 1     | 01    | uint8           |
| 3      |   [1]    | 2     | 02    | uint8           |
| 4      |   [2]    | 3     | 03    | uint8           |
|        | [1]      |       |       |                 |
| 5      |   [0]    | 4     | 04    | uint8           |
| 6      |   [1]    | 5     | 05    | uint8           |
| 7      |   [2]    | 6     | 06    | uint8           |
+--------+----------+-------+-------+-----------------+
>>>
>>> unpack(multi_dim_array, bindata)
[[1, 2, 3], [4, 5, 6]]

Sized Arrays in Structures

See the Dimensioned Array Structure Member tutorial for information on creating structures with an array member and a second member for controlling its size (dimensions).

API Reference

class plum.array.ArrayX(fmt: Union[plum.transform.Transform, Type[plum.data.Data]], dims: Union[Sequence[Optional[int]], ArrayX, plum.int.IntX, None] = None, name: Optional[str] = None)

Array to bytes and bytes to array transform.

dims

Array dimensions.

fmt

Array element format.

name

Transform format name (for repr and dump “Format” column).

nbytes

Transform format size in bytes.

pack(value: Any) → bytes

Pack value as formatted bytes.

Raises:PackError if type error, value error, etc.
pack_and_dump(value: Any) → Tuple[bytes, plum.dump.Dump]

Pack value as formatted bytes and produce bytes summary.

Raises:PackError if type error, value error, etc.
unpack(buffer: bytes) → Any

Unpack value from formatted bytes.

Raises:UnpackError if insufficient bytes, excess bytes, or value error
unpack_and_dump(buffer: bytes) → Tuple[Any, plum.dump.Dump]

Unpack value from bytes and produce packed bytes summary.

Raises:UnpackError if insufficient bytes, excess bytes, or value error