QuickCrawl
Web Scraping API for AI Agents — Scrape, crawl, and map websites with a single binary.
QuickCrawl is a powerful Go-based web scraping service that brings intelligence to AI agents. Whether you're scraping a single page, crawling an entire website, or mapping out link structures — QuickCrawl handles the heavy lifting with a sophisticated multi-layered architecture combining HTTP fetching, browser automation, and LLM-powered structured extraction.
Features
| Feature | Description |
|---|---|
| Web Scraping | Convert any URL to Markdown, HTML, Plain Text, or Links |
| Async Crawling | BFS website crawler with depth/page limits and rate limiting |
| URL Mapping | Discover all URLs on a site instantly without scraping content |
| JavaScript Rendering | Auto-detect SPAs and render via LightPanda or Chrome |
| LLM Extraction | Send a JSON schema, get validated structured data back |
| Web Search | SearXNG-powered search for AI agent integration |
| MCP Server | Built-in stdio transport for seamless AI agent integration |
| Multi-format Output | Markdown, HTML, RawHTML, PlainText, Links, JSON |