Docs /Templates/ sinapsis_langchain_readers

FireCrawlLoaderWrapper

sinapsis_langchain_readersLangChaindocumentsdocument_loadingdynamicfiles

Python template · sinapsis_langchain_readers.templates.langchain_data_readers

Overview

Dynamic Template to load documents from Langchain document_loaders module The template loads the document either as a Document object in the generic_data field of DataContainer of each string as a TextPacket if add_document_as_text_packet is set as True

FireCrawlLoader — inherited documentation

FireCrawlLoader document loader integration

Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``.

.. code-block:: bash

pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key"

Instantiate: .. code-block:: python

from langchain_community.document_loaders import FireCrawlLoader

loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... )

Lazy load: .. code-block:: python

docs = [] docs_lazy = loader.lazy_load()

# async variant: # docs_lazy = await loader.alazy_load()

for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python

Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}

Async load: .. code-block:: python

docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata)

.. code-block:: python

Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}Initialize with API key and url.

Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py

Capabilities

Capabilities: not set

Installation

Install the published package using pip or uv.

Terminalbash
pip install sinapsis_langchain_readers==1.0.0
uv pip install sinapsis_langchain_readers==1.0.0
uv add sinapsis_langchain_readers==1.0.0

Configuration

Effective fields from the inspector schema, including nested Pydantic model fields.

metadata
sinapsis_core.template_base.base_models.TemplateMetadata | None
Optional
Default: null
firecrawlloader_initRequired
__init__Attributes
Required
Default: null
url
str
Optional
Default: null
mode
typing.Literal['crawl', 'scrape', 'map', 'extract']
Optional
Default: "crawl"
params
typing.Optional[dict]
Optional
Default: null
api_key
typing.Optional[str]
Optional
Default: null
api_url
typing.Optional[str]
Optional
Default: null
add_document_as_text_packet
Whether to add document as text packet or not. firecrawlloader_init url <class 'str'>: default: None api_key typing.Optional[str]: default: None api_url typing.Optional[str]: default: None mode typing.Literal['crawl',… Read full field description

Whether to add document as text packet or not. firecrawlloader_init url <class 'str'>: default: None api_key typing.Optional[str]: default: None api_url typing.Optional[str]: default: None mode typing.Literal['crawl', 'scrape', 'map', 'extract']: default: crawl params typing.Optional[dict]: default: None

bool
Optional
Default: false

Configuration example

Generated from documented defaults. Replace any placeholders and complete required nested settings before running.

yaml
templates:
  - template_name: "FireCrawlLoaderWrapper"
    class_name: "FireCrawlLoaderWrapper"
    attributes:
      firecrawlloader_init:
        url: null
        mode: "crawl"
        params: null
        api_key: null
        api_url: null
      add_document_as_text_packet: false
View original docstring example

The source example references WikipediaLoaderWrapper, not FireCrawlLoaderWrapper.

yaml
agent:
  name: my_test_agent
templates:
- template_name: InputTemplate
  class_name: InputTemplate
  attributes: {}
- template_name: WikipediaLoaderWrapper
  class_name: WikipediaLoaderWrapper    ## note that because it is a dynamic template,
  template_input: InputTemplate         ##  the class name depends on the actual class that is used.
  attributes:
    add_document_as_text_packet: false
    wikipedialoader_init:
        query: the query for wikipedia
        lang: en
        load_max_docs: 5000
        load_all_available_meta: False
        doc_content_chars_max: 4000

Inheritance

Immediate parent: sinapsis_langchain_readers.templates.langchain_data_readers.LangChainDataReaderBase

Environment variables

Environment definitions belong to their canonical Python modules.

SINAPSIS_BUILD_DOCS · defined elsewhere