@zack/robots-aware-crawl
Crawl a bounded set of pages while respecting robots.txt, depth limits, and URL filters.
Install
agentpm install @zack/robots-aware-crawl@0.1.0Load
import { load } from '@agentpm/sdk';
const t = await load('@zack/robots-aware-crawl@0.1.0');from agentpm import load
t = load("@zack/robots-aware-crawl@0.1.0")Weekly downloads
2
0%
Last publish
1m ago
v0.1.1
Security & integrity
- Package type: tool
- SHA256:
b19d…a24f - Author-signed: ed25519:p0BwB_FxcozP
- Registry-attested
- Malware: none
License
MITView on SPDX