Docs /architecture/ UDT Architecture
architecture / transport

UDT Architecture

The UDT data model, exchange semantics, payloads and transport independence.

udtdatatransportDocumentation latest

Universal Data Transport — Architecture

Overview

Universal Data Transport (UDT) is the overarching architecture and specification for representing, moving, and exchanging data between Sinapsis agents and other compatible systems.

UDT is broader than serialization. It defines:

  • the universal data model used by agents;

  • the semantic data contract used as data moves through agents and templates;

  • the representation of typed data as DataContainer and DataPacket objects;

  • the exchange of data between processes, services, and agents;

  • the wire representation used when data crosses a transport boundary;

  • payload codecs for efficiently representing specialized data;

  • artifact handling for payloads that should not be embedded directly in a message.

Standardize the data model and exchange semantics. Optimize the representation of individual payloads.

The UDT Architecture

◇ DiagramDrag nodes · Scroll to zoom · Drag canvas to pan
flowchart TB
    U[Universal Data Transport]
    U --> M[Data model: DataContainer / DataPacket / Payload]
    U --> S[Data semantics and exchange]
    U --> W[Versioned wire format]
    U --> C[Payload codecs]
    U --> A[Artifact storage and references]
    W --> T[Transport-independent bytes]

Green-bordered nodes reference Sinapsis templates. Hover or focus them to fetch documentation.

View Mermaid source
mermaid
flowchart TB
    U[Universal Data Transport]
    U --> M[Data model: DataContainer / DataPacket / Payload]
    U --> S[Data semantics and exchange]
    U --> W[Versioned wire format]
    U --> C[Payload codecs]
    U --> A[Artifact storage and references]
    W --> T[Transport-independent bytes]

The computation graph, Template transformations, and split/merge operations are specified in Agent Model and Template Model, not re-specified here.

Data Model

At the center of UDT is the DataContainer.

A DataContainer is the universal root data structure exchanged by UDT-compatible agents. It may contain multiple typed DataPacket objects, metadata, and nested structures.

text
DataContainer
├── DataPacket
│   └── Payload
├── DataPacket
│   └── Payload
└── ...

Agents communicate through DataContainer, not through arbitrary Python objects or arbitrary application-specific protocols.

Data Semantics

UDT defines the semantics of the data being processed and exchanged.

These semantics include:

  • the meaning of DataContainer and DataPacket;

  • packet contents and metadata;

  • annotations and enrichment;

  • transformations and derivations;

  • template inputs and outputs;

  • agent inputs and outputs;

  • movement of containers through a template graph;

  • exchange of containers between agents or other UDT participants.

Data semantics describe what the data means and how it participates in computation.

Serialization describes how that data is represented as bytes.

This separation allows the same semantic data model to exist independently of whether a container is:

  • being processed in memory by a template;

  • passed between templates inside an agent;

  • exchanged between agents;

  • serialized to disk;

  • sent over a network transport.

Exchange

An exchange is the movement of a DataContainer between UDT participants.

Participants may include agents, processes, services, workers, nodes, or external systems implementing UDT.

text
Agent A
   │
   │ UDT Exchange
   ▼
Agent B

The underlying transport may vary:

text
UDT Exchange
     │
     ├── HTTP
     ├── WebSocket
     ├── TCP
     ├── Message Queue
     ├── Shared Memory
     └── Other Transport

The exchange defines the data contract; the underlying transport defines how bytes are moved.

Wire Format

The Wire Format is the concrete serialized representation of a DataContainer when it crosses an exchange boundary.

text
DataContainer
      │
      ▼
UDT Wire Format
      │
      ▼
MessagePack
      │
      ▼
Transport

The current implementation uses MessagePack as the underlying serialization mechanism. MessagePack is an implementation detail of the wire representation, not the definition of UDT itself.

Framing and exchange envelopes

The Python protocol 0.1.0 document distinguishes the low-level MessagePack wire framing (protocol, major, minor, kind, payload) from higher-level exchange envelopes such as SSEEnvelope, which carry API/event context around an optional DataContainer. These are different layers.

The earlier UDT draft used the term Wire Envelope for the MessagePack framing itself. The Wire Format article uses the concrete implementation terminology so the two meanings are not mixed up.

Payload

A Payload is the actual content carried by a DataPacket.

Examples include:

text
NumPy array
Torch tensor
DataFrame
Image
Audio bytes
Source code
File
Structured application object

Payloads can use specialized Payload Codecs.

Payload Codecs

A Payload Codec is a representation used to encode and decode a particular class of payload.

Examples:

text
NumPy       → raw binary representation
Torch       → tensor binary representation
DataFrame   → Arrow IPC
Protobuf    → protobuf binary
Bytes       → raw bytes
File        → artifact reference

This lets UDT keep a universal container while allowing individual payload types to use efficient representations.

Standardize the container. Optimize the payload.

Protobuf Is a Payload Codec, Not a Second UDT Protocol

Protobuf may be useful for structured payloads that benefit from a compact schema-driven representation.

Within UDT, protobuf is therefore a payload codec. It does not replace DataContainer.

text
DataContainer
      │
      └── DataPacket
            │
            └── Protobuf Payload
                  │
                  └── protobuf bytes

Those bytes are then represented inside the UDT wire format.

An agent always receives a DataContainer; protobuf is simply one possible representation of one payload.

Why This Separation Matters

Without this separation, every specialized serialization technology can become its own protocol:

text
Agent A ── protobuf protocol ── Agent B
Agent A ── Arrow protocol ──── Agent C
Agent A ── custom binary ───── Agent D

UDT instead provides:

text
                     DataContainer
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
           Protobuf       Arrow       Binary
              │            │            │
              └────────────┼────────────┘
                           ▼
                      UDT Exchange

The external contract remains stable while individual payload representations can change independently.

Artifacts

Not every payload should be embedded directly into the wire message.

Large files or binary objects may instead be stored externally and represented by an ArtifactRef:

text
DataContainer
    │
    └── DataPacket
          │
          └── ArtifactRef
                ├── artifact_id
                ├── size
                ├── sha256
                ├── mime_type
                └── location

The wire message carries the reference; the artifact transport handles the actual object.

Serialization vs Exchange vs Transport

These terms should remain distinct.

Serialization

text
DataContainer
      ↓
Bytes

Wire Format

text
DataContainer
      ↓
Wire-format framing
      ↓
Serialized bytes

Exchange

text
Agent A
   ↓
Exchange
   ↓
Agent B

Transport

text
TCP / HTTP / WebSocket / Queue / etc.

Therefore:

text
DataContainer
     ↓
Serialization
     ↓
UDT Wire Format
     ↓
UDT Exchange
     ↓
Transport

UDT Nomenclature

TermMeaning
Universal Data Transport (UDT)The overall architecture and specification
DataContainerThe universal data structure processed and exchanged by agents and templates
DataPacketA typed unit of data contained within a DataContainer
PayloadThe actual content carried by a packet
TemplateA unit of computation that transforms a DataContainer into a DataContainer
AgentAn orchestrator composed of Templates organized as a graph, typically a DAG
Template GraphThe graph describing template execution and data flow within an Agent
Data SemanticsRules describing the meaning, transformation, enrichment, and flow of containers and packets
ExchangeThe movement of a DataContainer between UDT participants or system boundaries
Wire FormatThe serialized representation used for an exchange
Exchange envelopeAPI/event context around an optional DataContainer (e.g. SSEEnvelope)
Payload CodecA specialized representation for a particular payload type
ArtifactExternally stored data referenced by a packet
ArtifactStoreThe mechanism used to store and retrieve artifacts
TransportThe underlying mechanism that moves serialized bytes

Naming the Python Implementation

The architecture and implementation should be distinguishable.

The architecture can be referred to as:

Universal Data Transport (UDT)

The Python implementation can remain:

sinapsis-protocol

This leaves room for other implementations:

text
Universal Data Transport
        │
        ├── UDT Specification
        ├── UDT Wire Format
        └── Implementations
             ├── sinapsis-protocol (Python)
             ├── future Rust implementation
             ├── future Go implementation
             └── future TypeScript implementation

The protocol therefore becomes an interoperability specification rather than simply a Python serialization library.

Core Principle

UDT standardizes the data model and exchange contract while allowing specialized payload representations and transport mechanisms to evolve independently.

Published from Git revision 7b7ba25d627b