Docs /Templates/ sinapsis-huggingface-transformers

PaliGemmaInference

sinapsis-huggingface-transformersHuggingFacehuggingfaceimagepaligemmamodels

Python template · sinapsis_huggingface_transformers.templates.pali_gemma.pali_gemma_inference

Overview

Implementation of PaliGemma inference pipeline for image processing and caption generation.

This class handles the inference process for PaliGemma models, including image processing, caption generation, and annotation creation. It supports both basic captioning and detection/segmentation tasks.

Capabilities

Capabilities: not set

Installation

Install the published package using pip or uv.

Terminalbash
pip install sinapsis-huggingface-transformers==1.0.0
uv pip install sinapsis-huggingface-transformers==1.0.0
uv add sinapsis-huggingface-transformers==1.0.0

Configuration

Effective fields from the inspector schema, including nested Pydantic model fields.

device
typing.Literal['cuda', 'cpu']
Optional
Default: "cpu"
prompt
Prompt to run the inference (default: "<image>caption en") The <image> token is essential as it serves as a marker that tells the model where to look at the image when processing the input. This token enables the model… Read full field description

Prompt to run the inference (default: "<image>caption en") The <image> token is essential as it serves as a marker that tells the model where to look at the image when processing the input. This token enables the model to understand the relationship between the visual and textual components during processing. Example prompts: - "<image>caption en" -> Generates a basic caption in English - "<image>What objects can you see in this image?" -> Lists objects in the image

str
Optional
Default: "<image>caption en"
metadata
sinapsis_core.template_base.base_models.TemplateMetadata | None
Optional
Default: null
model_path
typing.Literal['google/paligemma2-3b-pt-224']
Optional
Default: "google/paligemma2-3b-pt-224"
torch_dtype
typing.Literal['float16', 'float32']
Optional
Default: "float16"
max_new_tokens
int
Optional
Default: 200
processor_pathRequired
str
Required
Default: null
model_cache_dir
str
Optional
Default: "/home/ariel/.cache/sinapsis"

Configuration example

Example from the Python docstring:

yaml
agent:
  name: my_test_agent
templates:
- template_name: InputTemplate
  class_name: InputTemplate
  attributes: {}
- template_name: PaliGemmaInference
  class_name: PaliGemmaInference
  template_input: InputTemplate
  attributes:
    model_path: '/path/to/paligemma/model'
    processor_path: '`/path/to/processor'
    model_cache_dir: /path/to/cache/dir
    device: 'cuda'
    max_new_tokens: 200
    torch_dtype: float16
    prompt: <image> caption en
Show schema-based configuration skeleton
yaml
templates:
  - template_name: "PaliGemmaInference"
    class_name: "PaliGemmaInference"
    attributes:
      device: "cpu"
      prompt: "<image>caption en"
      model_path: "google/paligemma2-3b-pt-224"
      torch_dtype: "float16"
      max_new_tokens: 200
      processor_path: "<required>"
      model_cache_dir: "/home/ariel/.cache/sinapsis"

Inheritance

Immediate parent: sinapsis_huggingface_transformers.templates.pali_gemma.pali_gemma_base.PaliGemmaBase

Environment variables

Environment definitions belong to their canonical Python modules.

No directly related environment variables listed.