Six archives in parallel
Wayback Machine, Common Crawl, OTX, VirusTotal, URLScan and GitHub Code Search.
All projects
OSINT URL Collector · Rust
Urx
Collect URLs for a domain from OSINT archives — Wayback, Common Crawl, OTX and more — in parallel, then filter, normalize and dedupe. Built in Rust for speed.
Rust
170+
MIT
6
Data sources
3
Output formats
170+
GitHub stars
MIT
License
Capabilities
Map a domain's URL surface, fast
Async fan-out across six archives, with caching and incremental scans so repeat runs only surface what is new.
API key rotation
Rotate multiple provider keys to push past per-key rate limits.
Filter and presets
Narrow results by extension, pattern or preset (for example, no-images).
Normalize and dedupe
URL normalization and endpoint merging collapse noisy, near-duplicate URLs.
Status and links
Validate HTTP status codes and extract further links from responses.
Caching and incremental
Local SQLite or remote Redis cache enables fast incremental rescans.
Quickstart
One domain to a clean URL list
$ cargo install urx
# collect every archived URL for a domain
$ urx example.com -o results.txt
[*] 12,480 URLs from 6 sources
# JavaScript only, normalized and deduped
$ urx example.com -e js --normalize-url --merge-endpoint -f json
# only what's new since last run
$ urx example.com --incremental --cache-type sqlite
Illustrative output.