Docs /Templates/ sinapsis-llama-cpp

LLaMACPPTextCompletion

sinapsis-llama-cppLLaMA.cppchatbotscontextllamalarge_language_models

Python template · sinapsis_llama_cpp.templates.llama_cpp_text_completion

Overview

Template for llama-cpp text completion using `LLMConversationPacket`.

Capabilities

Capabilities: not set

Installation

Install the published package using pip or uv.

Terminalbash
pip install sinapsis-llama-cpp==1.0.0
uv pip install sinapsis-llama-cpp==1.0.0
uv add sinapsis-llama-cpp==1.0.0

Configuration

Effective fields from the inspector schema, including nested Pydantic model fields.

metadata
sinapsis_core.template_base.base_models.TemplateMetadata | None
Optional
Default: null
init_argsRequired

Llama runtime initialization arguments.

LLaMACPPInitArgs
Required
Default: null
seed

Base RNG seed. Defaults to `LLAMA_DEFAULT_SEED`.

int
Optional
Default: 4294967295
n_ctx

Context window size. Defaults to `512`.

int
Optional
Default: 512
n_batch

Prompt/prefill batch size. Defaults to `2048`.

int
Optional
Default: 2048
verbose

Verbose llama.cpp logging. Defaults to `True`.

bool
Optional
Default: true
main_gpu

Primary GPU index. Defaults to `0`.

int
Optional
Default: 0
n_ubatch

Micro-batch size. Defaults to `512`.

int
Optional
Default: 512
use_mmap

Whether to memory-map the model file. Defaults to `True`.

bool
Optional
Default: true
lora_base

Optional base model path for LoRA setups. Defaults to `None`.

str | None
Optional
Default: null
lora_path

LoRA adapter path. Defaults to `None`.

str | None
Optional
Default: null
n_threads

CPU threads for generation. Defaults to `None`.

int | None
Optional
Default: null
use_mlock

Whether to lock the model in RAM. Defaults to `False`.

bool
Optional
Default: false
lora_scale

LoRA strength. Defaults to `1.0`.

float
Optional
Default: 1
split_mode

Multi-GPU split mode (0=none, 1=row, 2=layer). Defaults to `0`.

typing.Literal[0, 1, 2]
Optional
Default: 0
chat_format

Chat template id used by `create_chat_completion`. Defaults to `None`.

str | None
Optional
Default: null
offload_kqv

Whether to offload KV/attention tensors to GPU. Defaults to `False`.

bool
Optional
Default: false
n_gpu_layers

Layers to offload to GPU (0=CPU, -1=all). Defaults to `0`.

int
Optional
Default: 0
tensor_split

Per-GPU split fractions. Defaults to `None`.

list[float] | None
Optional
Default: null
llm_model_fileRequired

GGUF filename to load.

str
Required
Default: null
llm_model_nameRequired

Model repo/name or local identifier to load from.

str
Required
Default: null
flash_attn_type

Flash-attention policy. Defaults to `-1`.

typing.Literal[-1, 0, 1]
Optional
Default: -1
n_threads_batch

CPU threads for prompt eval. Defaults to `None`.

int | None
Optional
Default: null
completion_args

Llama generation arguments.

LLaMACPPCompletionArgs
Optional
Default: null
seed

Per-request seed override. Defaults to `None`.

int | None
Optional
Default: null
stop

Stop sequence(s). Defaults to `None`.

str | list[str] | None
Optional
Default: null
min_p

Minimum probability threshold. Defaults to `0.05`.

float
Optional
Default: 0.05
top_k
int | None
Optional
Default: null
top_p
float | None
Optional
Default: null
typical_p

Typical sampling value. Defaults to `1.0`.

float
Optional
Default: 1
logit_bias

Token logit bias overrides. Defaults to `None`.

dict[int, float] | None
Optional
Default: null
max_tokensRequired

Maximum number of output tokens.

int
Required
Default: null
temperature
float | None
Optional
Default: null
penalty_last_n

Repetition window. Defaults to `64`.

int
Optional
Default: 64
repeat_penalty

Repetition penalty. Defaults to `1.0`.

float
Optional
Default: 1
present_penalty

Presence penalty. Defaults to `0.0`.

float
Optional
Default: 0
response_format

LLaMA structured-output configuration.

LLaMACPPResponseFormat
Optional
Default: null
type_
typing.Literal['text', 'json_object']
Optional
Default: "text"
schema_
SchemaDefinition
Optional
Default: null
required

Required root property names.

list[str]
Optional
Default: null
properties

Root schema properties.

dict[str, str | sinapsis_chatbots_base.helpers.schemas.PropertyDefinition]
Optional
Default: null
additional_properties

Whether extra root-level fields are allowed.

bool | None
Optional
Default: null
frequency_penalty

Frequency penalty. Defaults to `0.0`.

float
Optional
Default: 0
cleanup_response
bool
Optional
Default: true
reasoning_end_tag
str | None
Optional
Default: "</think>"
reasoning_start_tag
str | None
Optional
Default: "<think>"

Configuration example

Example from the Python docstring:

yaml
agent:
  name: my_test_agent
templates:
- template_name: InputTemplate
  class_name: InputTemplate
  attributes: {}
- template_name: LLaMACPPTextCompletion
  class_name: LLaMACPPTextCompletion
  template_input: InputTemplate
  attributes:
    init_args:
      llm_model_name: bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF
      llm_model_file: DeepSeek-R1-Distill-Qwen-7B-Q4_K_S.gguf
      n_ctx: 8192
      n_threads: 8
      n_gpu_layers: 15
      flash_attn_type: -1
      seed: 10
    completion_args:
      max_tokens: 4096
      temperature: 0.2
      seed: 10
Show schema-based configuration skeleton
yaml
templates:
  - template_name: "LLaMACPPTextCompletion"
    class_name: "LLaMACPPTextCompletion"
    attributes:
      init_args:
        seed: 4294967295
        n_ctx: 512
        n_batch: 2048
        verbose: true
        main_gpu: 0
        n_ubatch: 512
        use_mmap: true
        lora_base: null
        lora_path: null
        n_threads: null
        use_mlock: false
        lora_scale: 1
        split_mode: 0
        chat_format: null
        offload_kqv: false
        n_gpu_layers: 0
        tensor_split: null
        llm_model_file: "<required>"
        llm_model_name: "<required>"
        flash_attn_type: -1
        n_threads_batch: null
      completion_args:
        seed: null
        stop: null
        min_p: 0.05
        top_k: null
        top_p: null
        typical_p: 1
        logit_bias: null
        max_tokens: "<required>"
        temperature: null
        penalty_last_n: 64
        repeat_penalty: 1
        present_penalty: 0
        response_format:
          type_: "text"
          schema_:
            required: null
            properties: null
            additional_properties: null
        frequency_penalty: 0
      cleanup_response: true
      reasoning_end_tag: "</think>"
      reasoning_start_tag: "<think>"

Inheritance

Immediate parent: sinapsis_chatbots_base.templates.llm_runtime_base.LLMRuntimeBase

Environment variables

Environment definitions belong to their canonical Python modules.

No directly related environment variables listed.