Enforce robots.txt Crawl-delay in URLFrontier queues - #2000

Merged

rzo1 merged 9 commits into

mainfrom

feat/1979-robots-delay

Jul 28, 2026

Commits on Jul 15, 2026

Commits on Jul 20, 2026

fix: robots pacing coalesces to the max observed delay, single endpoint only
dpol1
committed