EmbeddingNodeGenerator
Python template · sinapsis_llama_index.templates.embedding_node_generator
Overview
Generate embedded `TextNode` objects from text documents.
This template loads documents either from `container.texts` or from generic-data keys, splits them with a LlamaIndex text splitter, and attaches HuggingFace embeddings to the resulting nodes. The nodes are then stored in `container.generic_data[self.instance_name]` for later insertion into PGVector.
Capabilities
Capabilities: not setInstallation
Install the published package using pip or uv.
pip install sinapsis-llama-index==1.0.0uv pip install sinapsis-llama-index==1.0.0uv add sinapsis-llama-index==1.0.0Configuration
Effective fields from the inspector schema, including nested Pydantic model fields.
Optional
Default: null
The list of keys for retrieving the document data from the container. If `None`, it will try to use `container.texts`. Defaults to `None`.
Optional
Default: null
Configuration for the text splitter (e.g., chunk size, overlap).
Optional
Default: null
The separator string to use when joining text chunks. Defaults to `" "`.
Optional
Default: " "
The maximum size (in tokens or characters, depending on the splitter) of each chunk. Defaults to `DEFAULT_CHUNK_SIZE`.
Optional
Default: 1024
The number of tokens/characters to overlap between consecutive chunks. Defaults to `SENTENCE_CHUNK_OVERLAP`.
Optional
Default: 200
Optional
Default: true
The string used to identify paragraph breaks within the text. Defaults to `DEFAULT_PARAGRAPH_SEP`.
Optional
Default: "\n\n\n"
Optional
Default: true
A regex pattern used for secondary splitting within larger chunks. Defaults to `CHUNKING_REGEX`.
Optional
Default: "[^,.;。?!]+[,.;。?!]?|[,.;。?!]"
Configuration for the HuggingFace embedding model (e.g., model name, device).
Required
Default: null
The device to run the model on ('cuda', 'cpu', 'auto'). If 'auto', selects 'cuda' if available, otherwise 'cpu'. Defaults to `"auto"`.
Optional
Default: "auto"
Whether to normalize the embeddings. Defaults to `True`.
Optional
Default: true
Maximum sequence length for the model. Usually inferred if None. Defaults to `None`.
Optional
Default: null
The name or path of the HuggingFace embedding model (e.g., 'BAAI/bge-small-en-v1.5').
Required
Default: null
Path to cache directory for downloading/loading models. Defaults to `SINAPSIS_CACHE_DIR`.
Optional
Default: "/home/ariel/.cache/sinapsis"
Batch size for embedding generation. Defaults to `DEFAULT_EMBED_BATCH_SIZE`.
Optional
Default: 10
Whether to use parallel processing if multiple GPUs are available. Defaults to `False`.
Optional
Default: false
Instruction prefix for document embeddings (e.g., for Instructor models). Defaults to `None`.
Optional
Default: null
Instruction prefix for query embeddings (e.g., for Instructor models). Defaults to `None`.
Optional
Default: null
Allow execution of remote code for custom models. Defaults to `False`.
Optional
Default: false
Configuration example
Example from the Python docstring:
agent:
name: my_test_agent
templates:
- template_name: InputTemplate
class_name: InputTemplate
attributes: {}
- template_name: EmbeddingNodeGenerator
class_name: EmbeddingNodeGenerator
template_input: InputTemplate
attributes:
splitter_args:
chunk_size: 256
chunk_overlap: 32
separator: " "
embedding_config:
model_name: Snowflake/snowflake-arctic-embed-xs
normalize: true
embed_batch_size: 10
trust_remote_code: true
device: auto
parallel_process: falseShow schema-based configuration skeleton
templates:
- template_name: "EmbeddingNodeGenerator"
class_name: "EmbeddingNodeGenerator"
attributes:
generic_keys: null
splitter_args:
separator: " "
chunk_size: 1024
chunk_overlap: 200
include_metadata: true
paragraph_separator: "\n\n\n"
include_prev_next_rel: true
secondary_chunking_regex: "[^,.;。?!]+[,.;。?!]?|[,.;。?!]"
embedding_config:
device: "auto"
normalize: true
max_length: null
model_name: "<required>"
cache_folder: "/home/ariel/.cache/sinapsis"
embed_batch_size: 10
parallel_process: false
text_instruction: null
query_instruction: null
trust_remote_code: false
Inheritance
sinapsis_core.template_base.template.TemplateEnvironment variables
Environment definitions belong to their canonical Python modules.
No directly related environment variables listed.