UnslothInferenceInstruct
Python template · sinapsis_unsloth.templates.inference.unsloth_inference_instruct
Overview
Template for instruction-tuned models (e.g., Alpaca style).
Handles formatting of instruction prompts with optional context injection.
Capabilities
Capabilities: not setInstallation
Install the published package using pip or uv.
pip install sinapsis-unsloth==1.0.0uv pip install sinapsis-unsloth==1.0.0uv add sinapsis-unsloth==1.0.0Configuration
Effective fields from the inspector schema, including nested Pydantic model fields.
Optional
Default: false
Optional
Default: null
Required
Default: null
Data type for model weights. Defaults to "auto".
Optional
Default: "auto"
Directory where models are cached. Defaults to SINAPSIS_CACHE_DIR.
Optional
Default: "/home/ariel/.cache/sinapsis"
Device mapping strategy for model parallelism. Defaults to "sequential".
Optional
Default: "sequential"
The name of the model to be used.
Required
Default: null
Whether to use 4-bit quantization. Defaults to True.
Optional
Default: true
Whether to use 8-bit quantization. Defaults to False.
Optional
Default: false
Random state for reproducibility. Defaults to 3407.
Optional
Default: 3407
Whether to use 16-bit quantization. Defaults to False.
Optional
Default: false
Maximum rank for LoRA layers. Defaults to 64.
Optional
Default: 64
Whether to enable fast inference mode. Defaults to False.
Optional
Default: false
Maximum sequence length for input data. Defaults to 2048.
Optional
Default: 2048
Whether to perform full fine-tuning. Defaults to False.
Optional
Default: false
Maximum GPU memory utilization as a fraction of total available memory. Defaults to 0.5.
Optional
Default: 0.5
Gradient checkpointing strategy. Defaults to "unsloth".
Optional
Default: "unsloth"
Configuration object defining the task preamble (system instruction) to be prepended to the user input.
Required
Default: null
The system prompt or instruction preamble prepended to the user input (e.g., 'Below is an instruction...').
Required
Default: null
Required
Default: null
Minimum probability for a token to be considered. Defaults to None.
Optional
Default: null
The number of highest probability vocabulary tokens to keep for top-k-filtering. Defaults to 50.
Optional
Default: 50
If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. Defaults to 1.0.
Optional
Default: 1
Whether to use sampling; use greedy decoding otherwise. Defaults to True.
Optional
Default: true
Number of beams for beam search. 1 means no beam search. Defaults to 1.
Optional
Default: 1
Whether to use key/value cache for faster generation. Defaults to True.
Optional
Default: true
The value used to modulate the next token probabilities. 1.0 is neutral. Lower is more deterministic. Defaults to 1.0.
Optional
Default: 1
A string or list of strings that should stop generation when encountered. Defaults to None.
Optional
Default: null
The maximum number of tokens to generate.
Required
Default: null
The minimum number of tokens to generate. Defaults to None.
Optional
Default: null
The parameter for repetition penalty. 1.0 means no penalty. Defaults to 1.0.
Optional
Default: 1
The cache implementation to use. "dynamic" is standard, "static" pre-allocates memory (faster but higher VRAM), "offloaded" moves cache to CPU. Defaults to "dynamic".
Optional
Default: "dynamic"
Optional
Default: null
Configuration example
Example from the Python docstring:
agent:
name: my_test_agent
templates:
- template_name: InputTemplate
class_name: InputTemplate
attributes: {}
- template_name: UnslothInferenceInstruct
class_name: UnslothInferenceInstruct
template_input: InputTemplate
attributes:
model_args:
model_name: 'unsloth/Llama-3.2-3B-Instruct'
max_seq_length: 2048
dtype: auto
load_in_4bit: true
load_in_8bit: false
load_in_16bit: false
full_finetuning: false
device_map: sequential
use_gradient_checkpointing: unsloth
fast_inference: false
gpu_memory_utilization: 0.5
random_state: 3407
max_lora_rank: 64
rag_context_key: null
generate_args:
max_new_tokens: 256
min_new_tokens: null
stop_strings: null
do_sample: true
num_beams: 1
use_cache: true
cache_implementation: dynamic
temperature: 1.0
top_k: 50
top_p: 1.0
min_p: null
repetition_penalty: 1.0
stream: false
chat_config:
task_preamble: Write a response that appropriately completes the requestShow schema-based configuration skeleton
templates:
- template_name: "UnslothInferenceInstruct"
class_name: "UnslothInferenceInstruct"
attributes:
stream: false
model_args:
dtype: "auto"
cache_dir: "/home/ariel/.cache/sinapsis"
device_map: "sequential"
model_name: "<required>"
load_in_4bit: true
load_in_8bit: false
random_state: 3407
load_in_16bit: false
max_lora_rank: 64
fast_inference: false
max_seq_length: 2048
full_finetuning: false
gpu_memory_utilization: 0.5
use_gradient_checkpointing: "unsloth"
chat_config:
task_preamble: "<required>"
generate_args:
min_p: null
top_k: 50
top_p: 1
do_sample: true
num_beams: 1
use_cache: true
temperature: 1
stop_strings: null
max_new_tokens: "<required>"
min_new_tokens: null
repetition_penalty: 1
cache_implementation: "dynamic"
rag_context_key: null
Inheritance
sinapsis_unsloth.templates.inference.unsloth_inference_base.UnslothInferenceBaseEnvironment variables
Environment definitions belong to their canonical Python modules.
No directly related environment variables listed.