[plum] Quick Start¶
Numeric Types¶
plum
provides many basic types for packing and unpacking bytes.
For numeric types, they are offered in both big and little endian forms.
Consult the API reference pages for a complete summary. The following
demontrates simple integer packing and unpacking:
>>> from plum.int.big import UInt16
>>>
>>> UInt16.pack(1)
bytearray(b'\x00\x01')
>>>
>>> UInt16.unpack(b'\x00\x01') == 1
True
Arrays¶
Without further customization, Array
elements are UInt8
and
during byte unpacking greedily consume bytes until
exhaustion:
>>> from plum.array import Array
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class Greedy(Array, item_cls=UInt8):
... """Sample greedy array."""
...
>>> # greedy arrays accept any number of items in the constructor
>>> Greedy.pack([1, 2, 3])
bytearray(b'\x01\x02\x03')
>>>
>>> # greedy arrays consume everything that is left in the buffer
>>> Greedy.unpack(b'\x01\x02\x03\x04') == [1, 2, 3, 4]
True
Subclassing the Array
type offers the ability to control the
type of the items in the array as well as the array dimensions:
>>> class Array2x2(Array, dims=(2, 2), item_cls=UInt16):
... """Sample fixed sized (dimensioned) array."""
...
>>> Array2x2.pack([[1, 2], [3, 4]])
bytearray(b'\x00\x01\x00\x02\x00\x03\x00\x04')
>>>
>>> Array2x2.unpack(b'\x00\x01\x00\x02\x00\x03\x00\x04')
[[1, 2], [3, 4]]
Structures¶
A Structure
subclass offers a succinct and intuitive method for
defining a bytes layout for packing and unpacking structured data.
Within the subclass definition, add member definitions as class
attributes to specify the member type. Type annotations are optional.
Specifying a default value for the member facilitates instantiating
or packing without specifying the member and is optional. Structure
types support list
behaviors but also provide attribute
access to the members.
For example:
>>> from plum.structure import Member, Structure
>>> from plum.int.big import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
... byte: int = Member(cls=UInt8)
... word: int = Member(cls=UInt16, default=1)
...
>>> # use dictionaries to specify member values,
>>> # you may omit members with defaults
>>> MyStruct.pack({'byte': 2})
bytearray(b'\x02\x00\x01')
>>>
>>> # or construct instance with positional or keyword arguments,
>>> # you may omit members with defaults
>>> MyStruct(byte=2).pack()
bytearray(b'\x02\x00\x01')
>>>
>>> # unpack results in an instance of the class
>>> mystruct = MyStruct.unpack(b'\x02\x00\x01')
>>> mystruct
MyStruct(byte=2, word=1)
>>>
>>> # access members via attribute or index
>>> mystruct.byte == 2
True
>>> mystruct[0] == 2
True
>>>
>>> # compare using list or dict
>>> mystruct == [2, 1], mystruct.asdict() == {'byte': 2, 'word': 1}
(True, True)
>>>
>>> # or use instance to leverage member defaults
>>> mystruct == MyStruct(byte=2)
True
The Structure
type has many features. Be sure to consult the
API reference and Tutorials since you will likely use this
type often.
Byte Breakdown Summaries¶
plum
instances offer a dump
property to help visualize
the resulting byte structure, the associated value, and the method
of accessing the value. The property returns a dump object that
supports string conversion to facilitate logging and debugging as
well as supports calls to print the summary to the console.
For example:
>>> from plum.structure import Member, Structure
>>> from plum.int.little import UInt8, UInt16
>>>
>>> class MyStruct(Structure):
... byte: int = Member(cls=UInt8)
... word: int = Member(cls=UInt16)
...
>>> mystruct = MyStruct(byte=1, word=2)
>>> mystruct.dump()
+--------+-------------+-------+-------+----------+
| Offset | Access | Value | Bytes | Type |
+--------+-------------+-------+-------+----------+
| | | | | MyStruct |
| 0 | [0] (.byte) | 1 | 01 | UInt8 |
| 1 | [1] (.word) | 2 | 02 00 | UInt16 |
+--------+-------------+-------+-------+----------+
See also
Enumerations¶
The Enum
type supports creating subclasses for bytes packing/unpacking
that double as an integer enumeration. The following example
shows the improved representation during interactive inspection and within
the dump summary table:
>>> from plum.int.enum import Enum
>>> from plum.int.little import UInt8
>>> from plum.structure import Member, Structure
>>>
>>> class Pet(Enum, nbytes=1):
... CAT = 0
... DOG = 1
...
>>> class Family(Structure):
... nkids: int = Member(cls=UInt8)
... pet: Pet = Member(cls=Pet)
...
>>> family = Family.unpack(b'\x02\x01')
>>> family.dump()
+--------+--------------+---------+-------+--------+
| Offset | Access | Value | Bytes | Type |
+--------+--------------+---------+-------+--------+
| | | | | Family |
| 0 | [0] (.nkids) | 2 | 02 | UInt8 |
| 1 | [1] (.pet) | Pet.DOG | 01 | Pet |
+--------+--------------+---------+-------+--------+
>>>
>>> family.pet
<Pet.DOG: 1>
>>>
>>> family.pet is Pet.DOG
True
Bit Fields¶
Integer bit field (BitFields
) types resemble Structure
types. Both
provide mechanisms to pack or unpack sequences of bytes and conveniently
specify or access data within the bytes. But BitFields
types provide
access to individual bits and/or bit fields within the bytes buffer.
The example below demonstrates use of the BitField
class
to define the bit position and bit field size (in bits) for each
member in the bytes buffer. Notice, bit field access produces an
instance of the Python type from the member definition:
>>> from plum.int.bitfields import BitFields, BitField
>>>
>>> class MyBf(BitFields, nbytes=1):
... bflag: bool = BitField(pos=0, size=1)
... iflag: int = BitField(pos=1, size=1)
... nibble: int = BitField(pos=2, size=4)
... reserved: int = BitField(pos=6, size=2, default=0)
...
>>> # construct instance with keyword arguments,
>>> # fields with default may be omitted
>>> MyBf(bflag=True, iflag=0, nibble=15).pack().hex()
'3d'
>>> # or construct instance from integer specifying all bit fields at once
>>> MyBf(0x3d).pack().hex()
'3d'
>>> # unpack returns an instance
>>> mybf = MyBf.unpack(b'\x00')
>>> mybf.dump()
+--------+-----------+-------+-------+------+
| Offset | Access | Value | Bytes | Type |
+--------+-----------+-------+-------+------+
| 0 | | 0 | 00 | MyBf |
| [0] | .bflag | False | | bool |
| [1] | .iflag | 0 | | int |
| [2:5] | .nibble | 0 | | int |
| [6:7] | .reserved | 0 | | int |
+--------+-----------+-------+-------+------+
>>>
>>> # use named attributes to access fields
>>> mybf.bflag
False
>>> mybf.nibble
0
>>> # instances have int properties
mybf == 0
True
The BitFields
type offers a lot of flexibility and features. Visit the
BitFields Tutorial for more details.
Bit Flags¶
The Flag
type supports creating custom enumerations where each member
describes a boolean flag for the bits of interest within an integer.
The custom enumeration may then be used for packing/unpacking
bytes as an integer and provides improved access and representation of
the individual bits. For example:
>>> from plum.int.flag import Flag
>>>
>>> class Status16(Flag, nbytes=2, byteorder='big'):
... # member values are powers of two
... CARRY = 1
... OVERFLOW = 4
... PARITY = 8
...
>>> Status16.CARRY.pack()
bytearray(b'\x00\x01')
>>>
>>> status = Status16.unpack(b'\x00\x01')
>>> status
<Status16.CARRY: 1>
>>>
>>> # byte summary shows value for every bit field
>>> # and attribute name hint to access
>>> status.dump()
+--------+-----------+-------+-------+----------+
| Offset | Access | Value | Bytes | Type |
+--------+-----------+-------+-------+----------+
| 0 | | 1 | 00 01 | Status16 |
| [0] | .carry | True | | bool |
| [2] | .overflow | False | | bool |
| [3] | .parity | False | | bool |
+--------+-----------+-------+-------+----------+
>>>
>>> # access individual fields by named attribute
>>> status.overflow
False
>>> # bit operations retain type
>>> new_status = status | Status16.OVERFLOW
>>> new_status
<Status16.OVERFLOW|CARRY: 5>
Packing Items¶
The previous quick start examples utilized the pack()
method
to produce the byte sequence. Alternatively, the pack()
function
facilitates packing multiple items at once and mirrors the standard library
struct.pack()
function except instead of accepting a format string, it
accepts plum
types:
>>> from plum import pack
>>> from plum.int.big import UInt16, UInt8
>>>
>>> # format as single plum type accepts a value as positional argument
>>> fmt = UInt8
>>> pack(fmt, 1)
bytearray(b'\x01')
>>>
>>> # format as tuple/list of types accepts values as positional arguments
>>> fmt = (UInt8, UInt16)
>>> pack(fmt, 1, 2)
bytearray(b'\x01\x00\x02')
>>>
>>> # format as dictionary of types accepts values as keyword arguments
>>> fmt = {'a': UInt8, 'b': UInt16}
>>> pack(fmt, a=1, b=2)
bytearray(b'\x01\x00\x02')
The pack()
function supports packing arbitrarily nested values within
by specifying the structure using tuples/lists and dictionaries within the
format:
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> pack(fmt, a=(1, 2), b={'i': 3, 'ii': 4})
bytearray(b'\x01\x02\x03\x04')
Additional pack functions and methods exist for packing directly into buffers or obtaining byte summary dumps. Visit the Packing Tutorial for more details.
Unpacking Items¶
The previous quick start examples utilized the unpack()
method
to convert a byte sequence into a convenient Python form. Alternatively,
the unpack()
function facilitates unpacking multiple items at once
and mirrors the standard library struct.pack()
function except
instead of accepting a format string, it accepts plum
types:
>>> from plum import unpack
>>> from plum.int.big import UInt8
>>>
>>> # passing plum type unpacks a single item
>>> unpack(UInt8, b'\x01')
1
>>>
>>> # passing a tuple of plum types results in tuple of values
>>> unpack((UInt8, UInt8), b'\x01\x02')
(1, 2)
>>> # passing a dictionary of plum types results in dictionary of values
>>> unpack({'a': UInt8, 'b': UInt8}, b'\x03\x04')
{'a': 3, 'b': 4}
The unpack()
function supports organizing the unpacked values into an
arbitrarily nested structure by specifying the structure using tuples/lists
and dictionaries within the format:
>>> from pprint import pprint
>>>
>>> fmt = {'a': (UInt8, UInt8), 'b': {'i': UInt8, 'ii': UInt8}}
>>>
>>> unpack(fmt, b'\x01\x02\x03\x04')
{'a': (1, 2), 'b': {'i': 3, 'ii': 4}}
In some use cases such as when the number of items or the type of item
to unpack depends on previously unpacked items, items must be unpacked
one at a time. Each unpack operation consumes a piece, but not all
of the bytes. For these cases, house the data to unpack in a
Buffer
instance and use the unpack()
method.
>>> from plum import Buffer
>>> buffer = Buffer(b'\x00\x01\x99')
>>> buffer.unpack(UInt8)
0
>>> buffer.unpack(UInt8)
1
Use the buffer instance as a context manager to ensure that the unpacking operations
completely consumed the bytes. In the following example, the extra unconsumed byte
(0x99
) causes an exception:
>>> with Buffer(b'\x00\x01\x99') as buffer:
... item1 = buffer.unpack(UInt8)
... item2 = buffer.unpack(UInt8)
...
Traceback (most recent call last):
...
plum._exceptions.ExcessMemoryError: 1 unconsumed bytes
Additional unpack functions and methods exist for unpacking directly from a buffer or obtaining byte summary dumps. Visit the Unpacking Tutorial for more details.
Memory Views¶
Many plum
types support buffer “views”. A buffer view provides access
to buffer bytes. Instead of one time buffer interpretation that normal
pack()
and unpack()
offer, a view provides dynamic
interpretation every time it is used. To create a buffer view, call
the view()
method passing it the buffer instance, and the offset
into the buffer in which to interpret. Then either use the view directly or
use the get()
or set()
methods to
access. For example:
>>> from plum.int.big import UInt8
>>>
>>> buffer = bytearray([1, 99, 3, 4])
>>>
>>> view = UInt8.view(buffer, offset=1)
>>> view
<view at 0x1: UInt8(99)>
>>>
>>> # use directly
>>> view == 99
True
>>>
>>> # obtain unpacked buffer bytes (same result as unpack() would give)
>>> view.get()
99
>>>
>>> # set new value in buffer
>>> view.set(0)
>>> buffer # notice byte at offset 1 changed
bytearray(b'\x01\x00\x03\x04')
>>>
>>> # continued usage of original view reflects updated buffer byte
>>> view == 0
True
>>>
>>> # in-place operations on views alter buffer bytes as well
>>> view += 0x99
>>> buffer # notice byte at offset 1 changed back
bytearray(b'\x01\x99\x03\x04')
The examples above show using a view for accessing shared buffers. Views also work
great for accessing remote or shared memory. For those applications, write your
own custom bytes buffer class that supports index operations (the __getitem__
and __setitem__
special methods). Unlike standard plum
types, views support
pointer de-referencing.