Docs /Templates/ sinapsis_langchain_readers

ArxivLoaderWrapper

sinapsis_langchain_readersLangChaindocumentsdocument_loadingdynamicfiles

Python template · sinapsis_langchain_readers.templates.langchain_data_readers

Overview

Dynamic Template to load documents from Langchain document_loaders module The template loads the document either as a Document object in the generic_data field of DataContainer of each string as a TextPacket if add_document_as_text_packet is set as True

ArxivLoader — inherited documentation

Load a query result from `Arxiv`. The loader converts the original PDF format into the text.

Setup: Install ``arxiv`` and ``PyMuPDF`` packages. ``PyMuPDF`` transforms PDF files downloaded from the arxiv.org site into the text format.

.. code-block:: bash

pip install -U arxiv pymupdf

Instantiate: .. code-block:: python

from langchain_community.document_loaders import ArxivLoader

loader = ArxivLoader( query="reasoning", # load_max_docs=2, # load_all_available_meta=False )

Load: .. code-block:: python

docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggre { 'Published': '2024-02-29', 'Title': 'Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation', 'Authors': 'Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, William Yang Wang', 'Summary': 'Pre-trained language models (LMs) are able to perform complex reasoning without explicit fine-tuning...' }

Lazy load: .. code-block:: python

docs = [] docs_lazy = loader.lazy_load()

# async variant: # docs_lazy = await loader.alazy_load()

for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python

Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggre { 'Published': '2024-02-29', 'Title': 'Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation', 'Authors': 'Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, William Yang Wang', 'Summary': 'Pre-trained language models (LMs) are able to perform complex reasoning without explicit fine-tuning...' }

Async load: .. code-block:: python

docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python

Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggre { 'Published': '2024-02-29', 'Title': 'Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation', 'Authors': 'Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, William Yang Wang', 'Summary': 'Pre-trained language models (LMs) are able to perform complex reasoning without explicit fine-tuning...' }

Use summaries of articles as docs: .. code-block:: python

from langchain_community.document_loaders import ArxivLoader

loader = ArxivLoader( query="reasoning" )

docs = loader.get_summaries_as_docs() print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python

Pre-trained language models (LMs) are able to perform complex reasoning without explicit fine-tuning { 'Entry ID': 'http://arxiv.org/abs/2402.03268v2', 'Published': datetime.date(2024, 2, 29), 'Title': 'Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation', 'Authors': 'Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, William Yang Wang' }Initialize with search query to find documents in the Arxiv. Supports all arguments of `ArxivAPIWrapper`.

Args: query: free text which used to find documents in the Arxiv doc_content_chars_max: cut limit for the length of a document's content

Capabilities

Capabilities: not set

Installation

Install the published package using pip or uv.

Terminalbash
pip install sinapsis_langchain_readers==1.0.0
uv pip install sinapsis_langchain_readers==1.0.0
uv add sinapsis_langchain_readers==1.0.0

Configuration

Effective fields from the inspector schema, including nested Pydantic model fields.

metadata
sinapsis_core.template_base.base_models.TemplateMetadata | None
Optional
Default: null
arxivloader_initRequired
BaseModel
Required
Default: null
add_document_as_text_packet
Whether to add document as text packet or not. arxivloader_init!!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. Attributes: __class_vars__: The names of the… Read full field description

Whether to add document as text packet or not. arxivloader_init!!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. Attributes: __class_vars__: The names of the class variables defined on the model. __private_attributes__: Metadata about the private attributes of the model. __signature__: The synthesized `__init__` [`Signature`][inspect.Signature] of the model. __pydantic_complete__: Whether model building is completed, or if there are still undefined fields. __pydantic_core_schema__: The core schema of the model. __pydantic_custom_init__: Whether the model has a custom `__init__` function. __pydantic_decorators__: Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. __pydantic_generic_metadata__: A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. __pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models. __pydantic_post_init__: The name of the post-init method for the model, if defined. __pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. __pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model. __pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model. __pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. __pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. __pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. __pydantic_fields_set__: The names of fields explicitly set during instantiation. __pydantic_private__: Values of private attributes set on the model instance.

bool
Optional
Default: false

Configuration example

Generated from documented defaults. Replace any placeholders and complete required nested settings before running.

yaml
templates:
  - template_name: "ArxivLoaderWrapper"
    class_name: "ArxivLoaderWrapper"
    attributes:
      arxivloader_init: {}
      add_document_as_text_packet: false
View original docstring example

The source example references WikipediaLoaderWrapper, not ArxivLoaderWrapper.

yaml
agent:
  name: my_test_agent
templates:
- template_name: InputTemplate
  class_name: InputTemplate
  attributes: {}
- template_name: WikipediaLoaderWrapper
  class_name: WikipediaLoaderWrapper    ## note that because it is a dynamic template,
  template_input: InputTemplate         ##  the class name depends on the actual class that is used.
  attributes:
    add_document_as_text_packet: false
    wikipedialoader_init:
        query: the query for wikipedia
        lang: en
        load_max_docs: 5000
        load_all_available_meta: False
        doc_content_chars_max: 4000

Inheritance

Immediate parent: sinapsis_langchain_readers.templates.langchain_data_readers.LangChainDataReaderBase

Environment variables

Environment definitions belong to their canonical Python modules.

SINAPSIS_BUILD_DOCS · defined elsewhere