Docs /architecture/ Protocol Implementation
architecture / transport

Protocol Implementation

Python sinapsis-protocol 0.1.0 architecture, modules and scope.

protocolpythonimplementationDocumentation latest

Sinapsis Protocol Implementation

sinapsis-protocol is the Python implementation of the UDT Architecture. These notes describe the source document's 0.1.0 design, not a verification against a live release.

Architecture

◇ DiagramDrag nodes · Scroll to zoom · Drag canvas to pan
flowchart TD
    A[Sinapsis DataContainer] --> B[Recursive dataclass walker]
    B --> C[Nested DataPackets / dataclasses]
    C --> D{Payload}

    D -->|NumPy| E[NumPy payload codec]
    D -->|Torch| F[Torch payload codec]
    D -->|Pandas / Polars| G[Arrow payload codec]
    D -->|bytes / string / path| H[Core payload representation]
    D -->|Protobuf / custom type| I[Registered payload codec]

    E --> J[Encoded payload]
    F --> J
    G --> J
    H --> J
    I --> J

    J --> K[DataContainer wire representation]

    K --> L{Payload size}
    L -->|small| M[Inline payload]
    L -->|large| N[ArtifactRef]

    M --> O[Versioned MessagePack wire format]
    N --> O

    O --> P[Wire representation]
    P --> Q[Transport]

    Q --> R[Deserializer]
    R --> S[Trusted dataclass / payload codec registry]
    S --> T[Reconstructed DataContainer]

 

Green-bordered nodes reference Sinapsis templates. Hover or focus them to fetch documentation.

View Mermaid source
mermaid
flowchart TD
    A[Sinapsis DataContainer] --> B[Recursive dataclass walker]
    B --> C[Nested DataPackets / dataclasses]
    C --> D{Payload}

    D -->|NumPy| E[NumPy payload codec]
    D -->|Torch| F[Torch payload codec]
    D -->|Pandas / Polars| G[Arrow payload codec]
    D -->|bytes / string / path| H[Core payload representation]
    D -->|Protobuf / custom type| I[Registered payload codec]

    E --> J[Encoded payload]
    F --> J
    G --> J
    H --> J
    I --> J

    J --> K[DataContainer wire representation]

    K --> L{Payload size}
    L -->|small| M[Inline payload]
    L -->|large| N[ArtifactRef]

    M --> O[Versioned MessagePack wire format]
    N --> O

    O --> P[Wire representation]
    P --> Q[Transport]

    Q --> R[Deserializer]
    R --> S[Trusted dataclass / payload codec registry]
    S --> T[Reconstructed DataContainer]

 

Note: Protobuf and other specialized formats are payload codecs. They encode individual payloads inside DataPackets; they do not bypass the UDT container or wire format.

The generic path is:

text
DataContainer
    ↓
recursive dataclass walker
    ↓
nested packet dataclasses
    ↓
leaf dispatch
    ↓
typed payload representation / inline bytes / ArtifactRef
    ↓
versioned MessagePack envelope

A specialized application can intentionally bypass this path:

text
Application contract
    ↓
protobuf schema
    ↓
protobuf bytes

That is useful when two endpoints have a fixed, tightly controlled contract and do not need the generic DataContainer protocol.

Project layout

text
src/sinapsis_protocol/
├── __init__.py
├── container/
│   ├── serialize.py      # DataContainer integration
│   └── deserialize.py
├── wire/
│   ├── constants.py      # protocol vocabulary / enum-like values
│   ├── models.py         # strongly typed wire and payload models
│   ├── format.py         # MessagePack wire format and versioning
│   └── sentinels.py      # protocol-level sentinel states
├── dataclass/
│   ├── registry.py       # structured packet type registry
│   ├── encode.py         # recursive dataclass encoding
│   └── decode.py         # recursive dataclass decoding
├── codecs/
│   ├── base.py           # payload codec contract
│   ├── registry.py       # lazy codec registry and overrides
│   ├── numpy.py
│   ├── torch.py
│   ├── pandas.py
│   ├── polars.py
│   └── cv2.py
├── artifacts/
│   ├── base.py           # artifact storage contract
│   ├── ref.py            # ArtifactRef
│   └── local.py          # local artifact store
├── types.py              # shared protocol contracts
└── errors.py             # protocol-specific exceptions

What 0.1.0 does and does not promise

Included

  • recursive DataContainer/dataclass serialization;

  • MessagePack framing;

  • protocol and transport-type versions;

  • strongly typed internal protocol models;

  • built-in NumPy, Torch, Pandas, and Polars adapters;

  • inline and artifact-backed binary payloads;

  • configurable resource limits;

  • explicit custom dataclass registration;

  • explicit leaf-adapter overrides;

  • no implicit pickle fallback.

Deliberately outside the generic protocol

  • arbitrary Python object graphs;

  • live process state;

  • automatic serialization of arbitrary torch.nn.Module instances;

  • implicit pickle/cloudpickle execution;

  • a separate top-level protocol for each payload type;

  • application-specific fixed schemas replacing DataContainer.

Protobuf or another fixed schema can still be used inside a DataPacket when that payload benefits from it.

Development

Terminalbash
pip install -e '.[dev]'
pytest

The test suite should cover both protocol mechanics and real sinapsis_core integration before treating a release as production-ready.

Licensing

The original implementation document points to its project's LICENSE file. That file was not among the six supplied documents, so no license file or license terms are reproduced in this documentation repository. Check the implementation repository before reusing code outside its existing license.

Further reading

Published from Git revision 7b7ba25d627b