@zack/robots-aware-crawl
Crawl a bounded set of pages while respecting robots.txt, depth limits, and URL filters.
Install
agentpm install @zack/robots-aware-crawl@0.1.1Load
import { load } from '@agentpm/sdk';
const t = await load('@zack/robots-aware-crawl@0.1.1');Installs
8
All time
Stars
0
Unstarred
Latest stable
6m ago
v0.1.1
Security & integrity
- Package type: tool
- SHA256:
e3ae…5404 - Author-signed: ed25519:p0BwB_FxcozPFbY6
- Registry-attested
- Malware: none
License
MITView on SPDX