[plum] Quick Start¶

Numeric Types
Arrays
Structures
Byte Breakdown Summaries
Enumerations
Bit Fields
Bit Flags
Packing Items
Unpacking Items
Memory Views

Numeric Types ¶

plum provides many basic types for packing and unpacking bytes. For numeric types, they are offered in both big and little endian forms. Consult the API reference pages for a complete summary. The following demontrates simple integer packing and unpacking:

>>> from plum.int.big import UInt16
>>>
>>> UInt16.pack(1)
bytearray(b'\x00\x01')
>>>
>>> UInt16.unpack(b'\x00\x01') == 1
True

Arrays ¶

Without further customization, Array elements are UInt8 and during byte unpacking greedily consume bytes until exhaustion:

>>> from plum.array import Array
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class Greedy(Array, item_cls=UInt8):
...     """Sample greedy array."""
...
>>> # greedy arrays accept any number of items in the constructor
>>> Greedy.pack([1, 2, 3])
bytearray(b'\x01\x02\x03')
>>>
>>> # greedy arrays consume everything that is left in the buffer
>>> Greedy.unpack(b'\x01\x02\x03\x04') == [1, 2, 3, 4]
True

Subclassing the Array type offers the ability to control the type of the items in the array as well as the array dimensions:

>>> class Array2x2(Array, dims=(2, 2), item_cls=UInt16):
...      """Sample fixed sized (dimensioned) array."""
...
>>> Array2x2.pack([[1, 2], [3, 4]])
bytearray(b'\x00\x01\x00\x02\x00\x03\x00\x04')
>>>
>>> Array2x2.unpack(b'\x00\x01\x00\x02\x00\x03\x00\x04')
[[1, 2], [3, 4]]

Structures ¶

A Structure subclass offers a succinct and intuitive method for defining a bytes layout for packing and unpacking structured data. Within the subclass definition, add member definitions as class attributes to specify the member type. Type annotations are optional. Specifying a default value for the member facilitates instantiating or packing without specifying the member and is optional. Structure types support list behaviors but also provide attribute access to the members.

For example:

>>> from plum.structure import Member, Structure
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
...     byte: int = Member(cls=UInt8)
...     word: int = Member(cls=UInt16, default=1)
...
>>> # use dictionaries to specify member values,
>>> # you may omit members with defaults
>>> MyStruct.pack({'byte': 2})
bytearray(b'\x02\x00\x01')
>>>
>>> # or construct instance with positional or keyword arguments,
>>> # you may omit members with defaults
>>> MyStruct(byte=2).pack()
bytearray(b'\x02\x00\x01')
>>>
>>> # unpack results in an instance of the class
>>> mystruct = MyStruct.unpack(b'\x02\x00\x01')
>>> mystruct
MyStruct(byte=2, word=1)
>>>
>>> # access members via attribute or index
>>> mystruct.byte == 2
True
>>> mystruct[0] == 2
True
>>>
>>> # compare using list or dict
>>> mystruct == [2, 1], mystruct.asdict() == {'byte': 2, 'word': 1}
(True, True)
>>>
>>> # or use instance to leverage member defaults
>>> mystruct == MyStruct(byte=2)
True

The Structure type has many features. Be sure to consult the API reference and Tutorials since you will likely use this type often.

Byte Breakdown Summaries ¶

plum instances offer a dump property to help visualize the resulting byte structure, the associated value, and the method of accessing the value. The property returns a dump object that supports string conversion to facilitate logging and debugging as well as supports calls to print the summary to the console. For example:

>>> from plum.structure import Member, Structure
>>> from plum.int.little import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
...     byte: int = Member(cls=UInt8)
...     word: int = Member(cls=UInt16)
...
>>> mystruct = MyStruct(byte=1, word=2)
>>> mystruct.dump()
+--------+-------------+-------+-------+----------+
| Offset | Access      | Value | Bytes | Type     |
+--------+-------------+-------+-------+----------+
|        |             |       |       | MyStruct |
| 0      | [0] (.byte) | 1     | 01    | UInt8    |
| 1      | [1] (.word) | 2     | 02 00 | UInt16   |
+--------+-------------+-------+-------+----------+

Enumerations ¶

The Enum type supports creating subclasses for bytes packing/unpacking that double as an integer enumeration. The following example shows the improved representation during interactive inspection and within the dump summary table:

>>> from plum.int.enum import Enum
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, Structure
>>>
>>> class Pet(Enum, nbytes=1):
...     CAT = 0
...     DOG = 1
...
>>> class Family(Structure):
...     nkids: int = Member(cls=UInt8)
...     pet: Pet = Member(cls=Pet)
...
>>> family = Family.unpack(b'\x02\x01')
>>> family.dump()
+--------+--------------+---------+-------+--------+
| Offset | Access       | Value   | Bytes | Type   |
+--------+--------------+---------+-------+--------+
|        |              |         |       | Family |
| 0      | [0] (.nkids) | 2       | 02    | UInt8  |
| 1      | [1] (.pet)   | Pet.DOG | 01    | Pet    |
+--------+--------------+---------+-------+--------+
>>>
>>> family.pet
<Pet.DOG: 1>
>>>
>>> family.pet is Pet.DOG
True

Bit Fields ¶

Integer bit field (BitFields) types resemble Structure types. Both provide mechanisms to pack or unpack sequences of bytes and conveniently specify or access data within the bytes. But BitFields types provide access to individual bits and/or bit fields within the bytes buffer.

The example below demonstrates use of the BitField class to define the bit position and bit field size (in bits) for each member in the bytes buffer. Notice, bit field access produces an instance of the Python type from the member definition:

>>> from plum.int.bitfields import BitFields, BitField
>>>
>>> class MyBf(BitFields, nbytes=1):
...     bflag: bool = BitField(pos=0, size=1)
...     iflag: int = BitField(pos=1, size=1)
...     nibble: int = BitField(pos=2, size=4)
...     reserved: int = BitField(pos=6, size=2, default=0)
...
>>> # construct instance with keyword arguments,
>>> # fields with default may be omitted
>>> MyBf(bflag=True, iflag=0, nibble=15).pack().hex()
'3d'
>>> # or construct instance from integer specifying all bit fields at once
>>> MyBf(0x3d).pack().hex()
'3d'
>>> # unpack returns an instance
>>> mybf = MyBf.unpack(b'\x00')
>>> mybf.dump()
+--------+-----------+-------+-------+------+
| Offset | Access    | Value | Bytes | Type |
+--------+-----------+-------+-------+------+
| 0      |           | 0     | 00    | MyBf |
|  [0]   | .bflag    | False |       | bool |
|  [1]   | .iflag    | 0     |       | int  |
|  [2:5] | .nibble   | 0     |       | int  |
|  [6:7] | .reserved | 0     |       | int  |
+--------+-----------+-------+-------+------+
>>>
>>> # use named attributes to access fields
>>> mybf.bflag
False
>>> mybf.nibble
0
>>> # instances have int properties
mybf == 0
True

The BitFields type offers a lot of flexibility and features. Visit the BitFields Tutorial for more details.

Bit Flags ¶

The Flag type supports creating custom enumerations where each member describes a boolean flag for the bits of interest within an integer. The custom enumeration may then be used for packing/unpacking bytes as an integer and provides improved access and representation of the individual bits. For example:

>>> from plum.int.flag import Flag
>>>
>>> class Status16(Flag, nbytes=2, byteorder='big'):
...     # member values are powers of two
...     CARRY = 1
...     OVERFLOW = 4
...     PARITY = 8
...
>>> Status16.CARRY.pack()
bytearray(b'\x00\x01')
>>>
>>> status = Status16.unpack(b'\x00\x01')
>>> status
<Status16.CARRY: 1>
>>>
>>> # byte summary shows value for every bit field
>>> # and attribute name hint to access
>>> status.dump()
+--------+-----------+-------+-------+----------+
| Offset | Access    | Value | Bytes | Type     |
+--------+-----------+-------+-------+----------+
| 0      |           | 1     | 00 01 | Status16 |
|  [0]   | .carry    | True  |       | bool     |
|  [2]   | .overflow | False |       | bool     |
|  [3]   | .parity   | False |       | bool     |
+--------+-----------+-------+-------+----------+
>>>
>>> # access individual fields by named attribute
>>> status.overflow
False
>>> # bit operations retain type
>>> new_status = status | Status16.OVERFLOW
>>> new_status
<Status16.OVERFLOW|CARRY: 5>

Packing Items ¶

The previous quick start examples utilized the pack() method to produce the byte sequence. Alternatively, the pack() function facilitates packing multiple items at once and mirrors the standard library struct.pack() function except instead of accepting a format string, it accepts plum types:

>>> from plum import pack
>>> from plum.int.big import UInt16, UInt8
>>>
>>> # format as single plum type accepts a value as positional argument
>>> fmt = UInt8
>>> pack(fmt, 1)
bytearray(b'\x01')
>>>
>>> # format as tuple/list of types accepts values as positional arguments
>>> fmt = (UInt8, UInt16)
>>> pack(fmt, 1, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # format as dictionary of types accepts values as keyword arguments
>>> fmt = {'a': UInt8, 'b': UInt16}
>>> pack(fmt, a=1, b=2)
bytearray(b'\x01\x00\x02')

The pack() function supports packing arbitrarily nested values within by specifying the structure using tuples/lists and dictionaries within the format:

>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> pack(fmt, a=(1, 2), b={'i': 3, 'ii': 4})
bytearray(b'\x01\x02\x03\x04')

Additional pack functions and methods exist for packing directly into buffers or obtaining byte summary dumps. Visit the Packing Tutorial for more details.

Unpacking Items ¶

The previous quick start examples utilized the unpack() method to convert a byte sequence into a convenient Python form. Alternatively, the unpack() function facilitates unpacking multiple items at once and mirrors the standard library struct.pack() function except instead of accepting a format string, it accepts plum types:

>>> from plum import unpack
>>> from plum.int.big import UInt8
>>>
>>> # passing plum type unpacks a single item
>>> unpack(UInt8, b'\x01')
1
>>>
>>> # passing a tuple of plum types results in tuple of values
>>> unpack((UInt8, UInt8), b'\x01\x02')
(1, 2)
>>> # passing a dictionary of plum types results in dictionary of values
>>> unpack({'a': UInt8, 'b': UInt8}, b'\x03\x04')
{'a': 3, 'b': 4}

The unpack() function supports organizing the unpacked values into an arbitrarily nested structure by specifying the structure using tuples/lists and dictionaries within the format:

>>> from pprint import pprint
>>>
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> unpack(fmt, b'\x01\x02\x03\x04')
{'a': (1, 2), 'b': {'i': 3, 'ii': 4}}

In some use cases such as when the number of items or the type of item to unpack depends on previously unpacked items, items must be unpacked one at a time. Each unpack operation consumes a piece, but not all of the bytes. For these cases, house the data to unpack in a Buffer instance and use the unpack() method.

>>> from plum import Buffer
>>> buffer = Buffer(b'\x00\x01\x99')
>>> buffer.unpack(UInt8)
0
>>> buffer.unpack(UInt8)
1

Use the buffer instance as a context manager to ensure that the unpacking operations completely consumed the bytes. In the following example, the extra unconsumed byte (0x99) causes an exception:

>>> with Buffer(b'\x00\x01\x99') as buffer:
...     item1 = buffer.unpack(UInt8)
...     item2 = buffer.unpack(UInt8)
...
Traceback (most recent call last):
    ...
plum._exceptions.ExcessMemoryError: 1 unconsumed bytes

Additional unpack functions and methods exist for unpacking directly from a buffer or obtaining byte summary dumps. Visit the Unpacking Tutorial for more details.

Memory Views ¶

Many plum types support buffer “views”. A buffer view provides access to buffer bytes. Instead of one time buffer interpretation that normal pack() and unpack() offer, a view provides dynamic interpretation every time it is used. To create a buffer view, call the view() method passing it the buffer instance, and the offset into the buffer in which to interpret. Then either use the view directly or use the get() or set() methods to access. For example:

>>> from plum.int.big import UInt8
>>>
>>> buffer = bytearray([1, 99, 3, 4])
>>>
>>> view = UInt8.view(buffer, offset=1)
>>> view
<view at 0x1: UInt8(99)>
>>>
>>> # use directly
>>> view == 99
True
>>>
>>> # obtain unpacked buffer bytes (same result as unpack() would give)
>>> view.get()
99
>>>
>>> # set new value in buffer
>>> view.set(0)
>>> buffer  # notice byte at offset 1 changed
bytearray(b'\x01\x00\x03\x04')
>>>
>>> # continued usage of original view reflects updated buffer byte
>>> view == 0
True
>>>
>>> # in-place operations on views alter buffer bytes as well
>>> view += 0x99
>>> buffer  # notice byte at offset 1 changed back
bytearray(b'\x01\x99\x03\x04')

The examples above show using a view for accessing shared buffers. Views also work great for accessing remote or shared memory. For those applications, write your own custom bytes buffer class that supports index operations (the __getitem__ and __setitem__ special methods). Unlike standard plum types, views support pointer de-referencing.