Docs /Templates/ sinapsis-vllm

vLLMStreamingTextCompletion

sinapsis-vllmvLLMchatbotscontextlarge_language_modelstext_completion

Python template · sinapsis_vllm.templates.vllm_streaming_text_completion

Overview

Streaming version of the vLLM text completion template.

Capabilities

Capabilities: not set

Installation

Install the published package using pip or uv.

Terminalbash
pip install sinapsis-vllm==1.0.0
uv pip install sinapsis-vllm==1.0.0
uv add sinapsis-vllm==1.0.0

Configuration

Effective fields from the inspector schema, including nested Pydantic model fields.

metadata
sinapsis_core.template_base.base_models.TemplateMetadata | None
Optional
Default: null
init_argsRequired

vLLM engine configuration arguments.

vLLMInitArgs
Required
Default: null
seed

Random seed used by the engine.

int
Optional
Default: 0
dtype

Weight and activation dtype.

typing.Literal['auto', 'half', 'float16', 'bfloat16', 'float', 'float32']
Optional
Default: "auto"
download_dir

Directory where weights are downloaded and loaded from.

str
Optional
Default: "/home/ariel/.cache/sinapsis"
max_num_seqs

Maximum concurrent sequences.

int
Optional
Default: 256
quantization

Quantization method, when used.

typing.Optional[typing.Literal['awq', 'auto_awq', 'fp8', 'fbgemm_fp8', 'fp_quant', 'modelopt', 'modelopt_fp4', 'modelopt_mxfp8', 'modelopt_mixed', 'auto_gptq', 'gptq', 'gptq_marlin', 'awq_marlin', 'humming', 'compressed-tensors', 'experts_int8', 'quark', 'moe_wna16', 'torchao', 'inc', 'mxfp4', 'gpt_oss_mxfp4', 'deepseek_v4_fp8', 'online', 'fp8_per_tensor', 'fp8_per_block', 'fp8_per_channel', 'int8_per_channel_weight_only', 'nvfp4_per_token', 'mxfp8']]
Optional
Default: null
enforce_eager

Whether to disable CUDA graphs and force eager execution.

bool
Optional
Default: false
max_model_len

Maximum model context length.

int | None
Optional
Default: null
cpu_offload_gb

CPU offload size in GB.

float
Optional
Default: 0
llm_model_nameRequired
str
Required
Default: null
tokenizer_mode

Tokenizer mode.

typing.Literal['auto', 'hf', 'alpamayo2_super', 'mistral', 'deepseek_v32', 'deepseek_v4', 'inkling', 'kimi_k3', 'cohere']
Optional
Default: "auto"
disable_log_stats

Whether to disable periodic runtime stats logging.

bool
Optional
Default: false
trust_remote_code

Whether to allow custom model code.

bool
Optional
Default: false
tensor_parallel_size

Number of GPUs used for tensor parallelism.

int
Optional
Default: 1
gpu_memory_utilization

Fraction of GPU memory reserved for the model.

float
Optional
Default: 0.9
completion_args

vLLM sampling and generation arguments.

vLLMCompletionArgs
Optional
Default: null
seed

Optional per-request seed.

int | None
Optional
Default: null
stop

Stop sequences.

str | list[str] | None
Optional
Default: null
min_p

Minimum relative token probability.

float
Optional
Default: 0
top_k
int | None
Optional
Default: null
top_p
float | None
Optional
Default: null
bad_words

Words disallowed during generation.

list[str] | None
Optional
Default: null
ignore_eos

Whether EOS should be ignored.

bool
Optional
Default: false
max_tokens

Maximum number of generated tokens.

int
Optional
Default: 16
min_tokens

Minimum number of generated tokens.

int
Optional
Default: 0
temperature
float | None
Optional
Default: null
response_format

Structured output configuration.

vLLMResponseFormat
Optional
Default: null
type_
typing.Literal['text', 'json_object']
Optional
Default: "text"
schema_
SchemaDefinition
Optional
Default: null
required

Required root property names.

list[str]
Optional
Default: null
properties

Root schema properties.

dict[str, str | sinapsis_chatbots_base.helpers.schemas.PropertyDefinition]
Optional
Default: null
additional_properties

Whether extra root-level fields are allowed.

bool | None
Optional
Default: null
presence_penalty

Penalty for introducing new tokens.

float
Optional
Default: 0
frequency_penalty

Penalty based on token frequency.

float
Optional
Default: 0
repetition_penalty

Penalty based on token repetition.

float
Optional
Default: 1
cleanup_response
bool
Optional
Default: true
reasoning_end_tag
str | None
Optional
Default: "</think>"
reasoning_start_tag
str | None
Optional
Default: "<think>"

Configuration example

Example from the Python docstring:

yaml
agent:
      name: my_test_agent
    templates:
    - template_name: InputTemplate
      class_name: InputTemplate
      attributes: {}
    - template_name: vLLMStreamingTextCompletion
      class_name: vLLMStreamingTextCompletion
      template_input: InputTemplate
      attributes:
        init_args:
          llm_model_name: Qwen/Qwen3-1.7B
          max_model_len: 4096
        completion_args:
          temperature: 0.7
          max_tokens: 512
Show schema-based configuration skeleton
yaml
templates:
  - template_name: "vLLMStreamingTextCompletion"
    class_name: "vLLMStreamingTextCompletion"
    attributes:
      init_args:
        seed: 0
        dtype: "auto"
        download_dir: "/home/ariel/.cache/sinapsis"
        max_num_seqs: 256
        quantization: null
        enforce_eager: false
        max_model_len: null
        cpu_offload_gb: 0
        llm_model_name: "<required>"
        tokenizer_mode: "auto"
        disable_log_stats: false
        trust_remote_code: false
        tensor_parallel_size: 1
        gpu_memory_utilization: 0.9
      completion_args:
        seed: null
        stop: null
        min_p: 0
        top_k: null
        top_p: null
        bad_words: null
        ignore_eos: false
        max_tokens: 16
        min_tokens: 0
        temperature: null
        response_format:
          type_: "text"
          schema_:
            required: null
            properties: null
            additional_properties: null
        presence_penalty: 0
        frequency_penalty: 0
        repetition_penalty: 1
      cleanup_response: true
      reasoning_end_tag: "</think>"
      reasoning_start_tag: "<think>"

Inheritance

Immediate parent: sinapsis_chatbots_base.templates.llm_streaming_base.LLMStreamingTemplateBase

Environment variables

Environment definitions belong to their canonical Python modules.

HF_TOKEN · defined elsewhere