@zack/robots-aware-crawl
Crawl a bounded set of pages while respecting robots.txt, depth limits, and URL filters.
Install
agentpm install @zack/robots-aware-crawl@0.1.1Load
import { load } from '@agentpm/sdk';
const t = await load('@zack/robots-aware-crawl@0.1.1');from agentpm import load
t = load("@zack/robots-aware-crawl@0.1.1")Weekly downloads
1
0%
Last publish
1d ago
v0.1.1
Security & integrity
- SHA256:
e3ae…5404 - Author-signed: ed25519:p0BwB_FxcozP
- Registry-attested
- Malware: none
License
MITView on SPDX