Website Map API: Sub-Second Domain URL Discovery & Link Mapping
Discover all accessible URLs across any website in seconds. Fast, lightweight site mapping designed to feed downstream scraping, search indexing, and RAG pipelines.
Overview & Architectural Foundation
Map complete website link structures and extract all public URLs in seconds. Features sitemap parsing, link discovery, and hierarchy analysis without downloading full HTML.
The Map API (POST /api/v1/map) probes robots.txt, recursively inspects sitemap.xml registries, and executes shallow crawl sweeps to compile a comprehensive registry of live endpoints. Discovered URLs are validated, canonicalized, and returned in sorted hierarchical JSON.
Core Capabilities & Developer Features
Sub-Second Discovery Engine
Extract hundreds or thousands of page URLs in seconds by combining sitemap indexing with rapid link header probing.
Hierarchy & Subfolder Classification
Group discovered URLs into structured tree hierarchies categorized by subpaths, blogs, documentation, and product catalogs.
URL Pattern Filtering
Filter URLs using glob or regex patterns to isolate specific sections like /docs/*, /blog/*, or /products/*.
Low Cost & High Throughput
Map entire domains at a fraction of the cost and compute required to render and download full page bodies.
Architecture & Pipeline Execution Lifecycle
When integrating web scraping and crawling operations into production workflows, data integrity, anti-bot resilience, and predictable latency are paramount. GcrawlAI processes incoming extraction requests through an intelligent gateway that evaluates target domain security profiles, selects appropriate proxy routing topologies (datacenter, residential, or mobile), and assigns requests to isolated headless Chromium instances.
During page execution, dynamic DOM trees are rendered, client-side JavaScript challenges are solved autonomously, and layout hydration events are monitored. The resulting rendered HTML is passed through semantic sanitization heuristics that strip non-content elements, boilerplate navigation menus, cookie consent banners, and tracking scripts while strictly preserving structural headings, tabular data, code snippets, and canonical links.
For AI and LLM workflows, the sanitized document is converted into token-efficient Markdown or validated against your custom JSON Schema before being returned to your client application, ensuring that downstream vector embeddings and language model reasoning receive high-density factual context without noise.
Enterprise Reliability, Rate Limiting & Webhook Integration
GcrawlAI provides battle-tested infrastructure built for high concurrency and enterprise throughput. With automated proxy rotation, intelligent failure retries, and comprehensive telemetry across every endpoint, your engineering team can focus on building intelligent applications rather than maintaining fragile scraping scripts.
Production data ingestion pipelines require high throughput and robust error resilience. GcrawlAI APIs support both synchronous request-response patterns for low-latency interactive workflows and asynchronous batch processing for multi-thousand URL workloads. Automated exponential backoff with jitter is applied to transient network errors or rate-limited endpoints, ensuring maximum completion rates.
When executing large-scale crawl or batch jobs, real-time webhooks provide milestone progress updates and post-completion payloads directly to your ingestion endpoints or cloud storage buckets. All responses adhere to standardized schemas with transparent HTTP status codes, structured error payloads, and full execution metadata including render timings, proxy geography, and token metrics.
Frequently Asked Questions
How does the Map API differ from the Crawl API?
The Map API focuses exclusively on discovering and listing URLs quickly without rendering page content, making it 10x faster and cheaper when you only need a site index.
Can the Map API discover unlinked pages?
Yes, if those pages are published in the domain sitemap or RSS feeds.
How many URLs can the Map API return?
The Map API can return up to 50,000 URLs per domain request.