@zack/robots-aware-crawl
Crawl a bounded set of pages while respecting robots.txt, depth limits, and URL filters.
Install
agentpm install @zack/robots-aware-crawl@0.1.1Load
import { load } from '@agentpm/sdk';
const t = await load('@zack/robots-aware-crawl@0.1.1');from agentpm import load
t = load("@zack/robots-aware-crawl@0.1.1")Weekly downloads
1
0%
Last publish
1d ago
v0.1.1
Eval results (latest)
Coming Soon