FireCrawlLoaderExecuteNTimes
Python template · sinapsis_langchain_readers.templates.langchain_data_readers
Overview
Dynamic Template to load documents from Langchain document_loaders module The template loads the document either as a Document object in the generic_data field of DataContainer of each string as a TextPacket if add_document_as_text_packet is set as True
FireCrawlLoader — inherited documentation
FireCrawlLoader document loader integration
Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``.
.. code-block:: bash
pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key"
Instantiate: .. code-block:: python
from langchain_community.document_loaders import FireCrawlLoader
loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... )
Lazy load: .. code-block:: python
docs = [] docs_lazy = loader.lazy_load()
# async variant: # docs_lazy = await loader.alazy_load()
for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata)
.. code-block:: python
Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}
Async load: .. code-block:: python
docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata)
.. code-block:: python
Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []}Initialize with API key and url.
Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py
Capabilities
Capabilities: not setInstallation
Install the published package using pip or uv.
pip install sinapsis_langchain_readers==1.0.0uv pip install sinapsis_langchain_readers==1.0.0uv add sinapsis_langchain_readers==1.0.0Configuration
Effective fields from the inspector schema, including nested Pydantic model fields.
Optional
Default: null
Optional
Default: 1
Required
Default: null
Optional
Default: null
Optional
Default: "crawl"
Optional
Default: null
Optional
Default: null
Optional
Default: null
Optional
Default: false
Configuration example
Generated from documented defaults. Replace any placeholders and complete required nested settings before running.
templates:
- template_name: "FireCrawlLoaderExecuteNTimes"
class_name: "FireCrawlLoaderExecuteNTimes"
attributes:
n_times_execute: 1
firecrawlloader_init:
url: null
mode: "crawl"
params: null
api_key: null
api_url: null
add_document_as_text_packet: false
View original docstring example
The source example references WikipediaLoaderWrapper, not FireCrawlLoaderExecuteNTimes.
agent:
name: my_test_agent
templates:
- template_name: InputTemplate
class_name: InputTemplate
attributes: {}
- template_name: WikipediaLoaderWrapper
class_name: WikipediaLoaderWrapper ## note that because it is a dynamic template,
template_input: InputTemplate ## the class name depends on the actual class that is used.
attributes:
add_document_as_text_packet: false
wikipedialoader_init:
query: the query for wikipedia
lang: en
load_max_docs: 5000
load_all_available_meta: False
doc_content_chars_max: 4000,
n_times_execute determines the number of times the template's execute
method is called. By default, it only gets called once. Further calls
to execute are bypassed.Inheritance
sinapsis_langchain_readers.templates.langchain_data_readers.ExecuteNTimesLangchainDataReadersEnvironment variables
Environment definitions belong to their canonical Python modules.