UDT Architecture
The UDT data model, exchange semantics, payloads and transport independence.
Universal Data Transport — Architecture
Overview
Universal Data Transport (UDT) is the overarching architecture and specification for representing, moving, and exchanging data between Sinapsis agents and other compatible systems.
UDT is broader than serialization. It defines:
the universal data model used by agents;
the semantic data contract used as data moves through agents and templates;
the representation of typed data as
DataContainerandDataPacketobjects;the exchange of data between processes, services, and agents;
the wire representation used when data crosses a transport boundary;
payload codecs for efficiently representing specialized data;
artifact handling for payloads that should not be embedded directly in a message.
Standardize the data model and exchange semantics. Optimize the representation of individual payloads.
The UDT Architecture
flowchart TB
U[Universal Data Transport]
U --> M[Data model: DataContainer / DataPacket / Payload]
U --> S[Data semantics and exchange]
U --> W[Versioned wire format]
U --> C[Payload codecs]
U --> A[Artifact storage and references]
W --> T[Transport-independent bytes]
Green-bordered nodes reference Sinapsis templates. Hover or focus them to fetch documentation.
View Mermaid source
flowchart TB
U[Universal Data Transport]
U --> M[Data model: DataContainer / DataPacket / Payload]
U --> S[Data semantics and exchange]
U --> W[Versioned wire format]
U --> C[Payload codecs]
U --> A[Artifact storage and references]
W --> T[Transport-independent bytes]
The computation graph, Template transformations, and split/merge operations are specified in Agent Model and Template Model, not re-specified here.
Data Model
At the center of UDT is the DataContainer.
A DataContainer is the universal root data structure exchanged by UDT-compatible agents. It may contain multiple typed DataPacket objects, metadata, and nested structures.
DataContainer
├── DataPacket
│ └── Payload
├── DataPacket
│ └── Payload
└── ...
Agents communicate through
DataContainer, not through arbitrary Python objects or arbitrary application-specific protocols.
Data Semantics
UDT defines the semantics of the data being processed and exchanged.
These semantics include:
the meaning of
DataContainerandDataPacket;packet contents and metadata;
annotations and enrichment;
transformations and derivations;
template inputs and outputs;
agent inputs and outputs;
movement of containers through a template graph;
exchange of containers between agents or other UDT participants.
Data semantics describe what the data means and how it participates in computation.
Serialization describes how that data is represented as bytes.
This separation allows the same semantic data model to exist independently of whether a container is:
being processed in memory by a template;
passed between templates inside an agent;
exchanged between agents;
serialized to disk;
sent over a network transport.
Exchange
An exchange is the movement of a DataContainer between UDT participants.
Participants may include agents, processes, services, workers, nodes, or external systems implementing UDT.
Agent A
│
│ UDT Exchange
▼
Agent B
The underlying transport may vary:
UDT Exchange
│
├── HTTP
├── WebSocket
├── TCP
├── Message Queue
├── Shared Memory
└── Other Transport
The exchange defines the data contract; the underlying transport defines how bytes are moved.
Wire Format
The Wire Format is the concrete serialized representation of a DataContainer when it crosses an exchange boundary.
DataContainer
│
▼
UDT Wire Format
│
▼
MessagePack
│
▼
Transport
The current implementation uses MessagePack as the underlying serialization mechanism. MessagePack is an implementation detail of the wire representation, not the definition of UDT itself.
Framing and exchange envelopes
The Python protocol 0.1.0 document distinguishes the low-level MessagePack wire framing (protocol, major, minor, kind, payload) from higher-level exchange envelopes such as SSEEnvelope, which carry API/event context around an optional DataContainer. These are different layers.
The earlier UDT draft used the term Wire Envelope for the MessagePack framing itself. The Wire Format article uses the concrete implementation terminology so the two meanings are not mixed up.
Payload
A Payload is the actual content carried by a DataPacket.
Examples include:
NumPy array
Torch tensor
DataFrame
Image
Audio bytes
Source code
File
Structured application object
Payloads can use specialized Payload Codecs.
Payload Codecs
A Payload Codec is a representation used to encode and decode a particular class of payload.
Examples:
NumPy → raw binary representation
Torch → tensor binary representation
DataFrame → Arrow IPC
Protobuf → protobuf binary
Bytes → raw bytes
File → artifact reference
This lets UDT keep a universal container while allowing individual payload types to use efficient representations.
Standardize the container. Optimize the payload.
Protobuf Is a Payload Codec, Not a Second UDT Protocol
Protobuf may be useful for structured payloads that benefit from a compact schema-driven representation.
Within UDT, protobuf is therefore a payload codec. It does not replace DataContainer.
DataContainer
│
└── DataPacket
│
└── Protobuf Payload
│
└── protobuf bytes
Those bytes are then represented inside the UDT wire format.
An agent always receives a DataContainer; protobuf is simply one possible representation of one payload.
Why This Separation Matters
Without this separation, every specialized serialization technology can become its own protocol:
Agent A ── protobuf protocol ── Agent B
Agent A ── Arrow protocol ──── Agent C
Agent A ── custom binary ───── Agent D
UDT instead provides:
DataContainer
│
┌────────────┼────────────┐
▼ ▼ ▼
Protobuf Arrow Binary
│ │ │
└────────────┼────────────┘
▼
UDT Exchange
The external contract remains stable while individual payload representations can change independently.
Artifacts
Not every payload should be embedded directly into the wire message.
Large files or binary objects may instead be stored externally and represented by an ArtifactRef:
DataContainer
│
└── DataPacket
│
└── ArtifactRef
├── artifact_id
├── size
├── sha256
├── mime_type
└── location
The wire message carries the reference; the artifact transport handles the actual object.
Serialization vs Exchange vs Transport
These terms should remain distinct.
Serialization
DataContainer
↓
Bytes
Wire Format
DataContainer
↓
Wire-format framing
↓
Serialized bytes
Exchange
Agent A
↓
Exchange
↓
Agent B
Transport
TCP / HTTP / WebSocket / Queue / etc.
Therefore:
DataContainer
↓
Serialization
↓
UDT Wire Format
↓
UDT Exchange
↓
Transport
UDT Nomenclature
| Term | Meaning |
|---|---|
| Universal Data Transport (UDT) | The overall architecture and specification |
| DataContainer | The universal data structure processed and exchanged by agents and templates |
| DataPacket | A typed unit of data contained within a DataContainer |
| Payload | The actual content carried by a packet |
| Template | A unit of computation that transforms a DataContainer into a DataContainer |
| Agent | An orchestrator composed of Templates organized as a graph, typically a DAG |
| Template Graph | The graph describing template execution and data flow within an Agent |
| Data Semantics | Rules describing the meaning, transformation, enrichment, and flow of containers and packets |
| Exchange | The movement of a DataContainer between UDT participants or system boundaries |
| Wire Format | The serialized representation used for an exchange |
| Exchange envelope | API/event context around an optional DataContainer (e.g. SSEEnvelope) |
| Payload Codec | A specialized representation for a particular payload type |
| Artifact | Externally stored data referenced by a packet |
| ArtifactStore | The mechanism used to store and retrieve artifacts |
| Transport | The underlying mechanism that moves serialized bytes |
Naming the Python Implementation
The architecture and implementation should be distinguishable.
The architecture can be referred to as:
Universal Data Transport (UDT)
The Python implementation can remain:
sinapsis-protocol
This leaves room for other implementations:
Universal Data Transport
│
├── UDT Specification
├── UDT Wire Format
└── Implementations
├── sinapsis-protocol (Python)
├── future Rust implementation
├── future Go implementation
└── future TypeScript implementation
The protocol therefore becomes an interoperability specification rather than simply a Python serialization library.
Core Principle
UDT standardizes the data model and exchange contract while allowing specialized payload representations and transport mechanisms to evolve independently.
Published from Git revision 7b7ba25d627b