FireCrawlLoaderWrapper
Python template · sinapsis_langchain_readers.templates.langchain_data_readers
Overview
Dynamic Template to load documents from Langchain document_loaders module The template loads the document either as a Document object in the generic_data field of DataContainer of each string as a TextPacket if add_document_as_text_packet is set as True
FireCrawlLoader — inherited documentation
FireCrawlLoader document loader integration
Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``.
.. code-block:: bash
pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key"
Instantiate: .. code-block:: python
from langchain_community.document_loaders import FireCrawlLoader
loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... )
Lazy load: .. code-block:: python
docs = [] docs_lazy = loader.lazy_load()
# async variant: # docs_lazy = await loader.alazy_load()
for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata)
.. code-block:: python
Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}
Async load: .. code-block:: python
docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata)
.. code-block:: python
Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}Initialize with API key and url.
Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py
Capabilities
Capabilities: not setInstallation
Install the published package using pip or uv.
pip install sinapsis_langchain_readers==1.0.0uv pip install sinapsis_langchain_readers==1.0.0uv add sinapsis_langchain_readers==1.0.0Configuration
Effective fields from the inspector schema, including nested Pydantic model fields.
Optional
Default: null
Required
Default: null
Optional
Default: null
Optional
Default: "crawl"
Optional
Default: null
Optional
Default: null
Optional
Default: null
Whether to add document as text packet or not. firecrawlloader_init url <class 'str'>: default: None api_key typing.Optional[str]: default: None api_url typing.Optional[str]: default: None mode typing.Literal['crawl',… Read full field description
Whether to add document as text packet or not. firecrawlloader_init url <class 'str'>: default: None api_key typing.Optional[str]: default: None api_url typing.Optional[str]: default: None mode typing.Literal['crawl', 'scrape', 'map', 'extract']: default: crawl params typing.Optional[dict]: default: None
Optional
Default: false
Configuration example
Generated from documented defaults. Replace any placeholders and complete required nested settings before running.
templates:
- template_name: "FireCrawlLoaderWrapper"
class_name: "FireCrawlLoaderWrapper"
attributes:
firecrawlloader_init:
url: null
mode: "crawl"
params: null
api_key: null
api_url: null
add_document_as_text_packet: false
View original docstring example
The source example references WikipediaLoaderWrapper, not FireCrawlLoaderWrapper.
agent:
name: my_test_agent
templates:
- template_name: InputTemplate
class_name: InputTemplate
attributes: {}
- template_name: WikipediaLoaderWrapper
class_name: WikipediaLoaderWrapper ## note that because it is a dynamic template,
template_input: InputTemplate ## the class name depends on the actual class that is used.
attributes:
add_document_as_text_packet: false
wikipedialoader_init:
query: the query for wikipedia
lang: en
load_max_docs: 5000
load_all_available_meta: False
doc_content_chars_max: 4000Inheritance
sinapsis_langchain_readers.templates.langchain_data_readers.LangChainDataReaderBaseEnvironment variables
Environment definitions belong to their canonical Python modules.