HNLoaderWrapper
Python template · sinapsis_langchain_readers.templates.langchain_data_readers
Overview
Dynamic Template to load documents from Langchain document_loaders module The template loads the document either as a Document object in the generic_data field of DataContainer of each string as a TextPacket if add_document_as_text_packet is set as True
HNLoader — inherited documentation
Load `Hacker News` data.
It loads data from either main page results or the comments page.Initialize loader.
Args: web_paths: Web paths to load from. requests_per_second: Max number of concurrent requests to make. default_parser: Default parser to use for BeautifulSoup. requests_kwargs: kwargs for requests raise_for_status: Raise an exception if http status code denotes an error. bs_get_text_kwargs: kwargs for beatifulsoup4 get_text bs_kwargs: kwargs for beatifulsoup4 web page parsing show_progress: Show progress bar when loading pages. trust_env: set to True if using proxy to make web requests, for example using http(s)_proxy environment variables. Defaults to False.
Capabilities
Capabilities: not setInstallation
Install the published package using pip or uv.
pip install sinapsis_langchain_readers==1.0.0uv pip install sinapsis_langchain_readers==1.0.0uv add sinapsis_langchain_readers==1.0.0Configuration
Effective fields from the inspector schema, including nested Pydantic model fields.
Optional
Default: null
Required
Default: null
Optional
Default: null
Optional
Default: null
Optional
Default: null
Optional
Default: ""
Optional
Default: null
Optional
Default: false
Optional
Default: []
Optional
Default: true
Optional
Default: true
Optional
Default: "html.parser"
Optional
Default: null
Optional
Default: null
Optional
Default: true
Optional
Default: false
Optional
Default: null
Optional
Default: false
Optional
Default: 2
Whether to add document as text packet or not. hnloader_init web_path typing.Union[str, typing.Sequence[str]]: default: header_template typing.Optional[dict]: default: None verify_ssl <class 'bool'>: default: True… Read full field description
Whether to add document as text packet or not. hnloader_init web_path typing.Union[str, typing.Sequence[str]]: default: header_template typing.Optional[dict]: default: None verify_ssl <class 'bool'>: default: True proxies typing.Optional[dict]: default: None continue_on_failure <class 'bool'>: default: False autoset_encoding <class 'bool'>: default: True encoding typing.Optional[str]: default: None web_paths typing.Sequence[str]: default: () requests_per_second <class 'int'>: default: 2 default_parser <class 'str'>: default: html.parser requests_kwargs typing.Optional[typing.Dict[str, typing.Any]]: default: None raise_for_status <class 'bool'>: default: False bs_get_text_kwargs typing.Optional[typing.Dict[str, typing.Any]]: default: None bs_kwargs typing.Optional[typing.Dict[str, typing.Any]]: default: None session typing.Any: default: None show_progress <class 'bool'>: default: True trust_env <class 'bool'>: default: False
Optional
Default: false
Configuration example
Generated from documented defaults. Replace any placeholders and complete required nested settings before running.
templates:
- template_name: "HNLoaderWrapper"
class_name: "HNLoaderWrapper"
attributes:
hnloader_init:
proxies: null
session: null
encoding: null
web_path: ""
bs_kwargs: null
trust_env: false
web_paths: []
verify_ssl: true
show_progress: true
default_parser: "html.parser"
header_template: null
requests_kwargs: null
autoset_encoding: true
raise_for_status: false
bs_get_text_kwargs: null
continue_on_failure: false
requests_per_second: 2
add_document_as_text_packet: false
View original docstring example
The source example references WikipediaLoaderWrapper, not HNLoaderWrapper.
agent:
name: my_test_agent
templates:
- template_name: InputTemplate
class_name: InputTemplate
attributes: {}
- template_name: WikipediaLoaderWrapper
class_name: WikipediaLoaderWrapper ## note that because it is a dynamic template,
template_input: InputTemplate ## the class name depends on the actual class that is used.
attributes:
add_document_as_text_packet: false
wikipedialoader_init:
query: the query for wikipedia
lang: en
load_max_docs: 5000
load_all_available_meta: False
doc_content_chars_max: 4000Inheritance
sinapsis_langchain_readers.templates.langchain_data_readers.LangChainDataReaderBaseEnvironment variables
Environment definitions belong to their canonical Python modules.