Skip to content
Gridlode
Docs · Reference

Website Crawler (Markdown for RAG)

GET /v1/url/crawl1 credit / pagecache 60 min

Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word count and link depth. Built for RAG and LLM ingestion: navigation and boilerplate removed, duplicates dropped, very long pages truncated and flagged. Same-site only, include/exclude path patterns (glob or regex), optional JavaScript rendering in real Chrome. Polite by design: robots.txt and Crawl-delay respected, at most 4 requests in flight, hard page, byte and time limits. Charged per page returned (1 credit, 2 with rendering); failed crawls are free.

Query parameters

url
string · required
Start URL: a site root or any page. Only pages on the same site are crawled (same registrable domain, so www. and other subdomains are included).
limit
integer
Maximum number of pages to return. Charged per page returned: 1 credit, 2 with render=true.
Default: 10
max_depth
integer
How many links deep to follow from the start page (0 = only the start page and, with use_sitemap, the sitemap URLs).
Default: 2
include_paths
array
Only crawl URLs whose path matches one of these patterns. Glob (/blog/*, *.html), regex (prefix re: or start with ^), or a plain path prefix (/docs). Matched against the URL path and query. The start page is always read for links.
exclude_paths
array
Never crawl URLs whose path matches one of these patterns (exclude wins over include). Glob (/blog/*, *.html), regex (prefix re: or start with ^), or a plain path prefix (/docs). Matched against the URL path and query.
use_sitemap
boolean
Also seed the crawl with the URLs listed in the site's XML sitemap (found through robots.txt or /sitemap.xml).
Default: false
render
boolean
Render every page in real Chrome so JavaScript-built content is captured. Slower; 2 credits per page.
Default: false
include_links
boolean
Keep hyperlinks in the Markdown.
Default: false
respect_robots
boolean
Skip URLs disallowed by the site's robots.txt and honour its Crawl-delay.
Default: true
max_age
integer
Accept a cached answer up to this many seconds old, billed at 50%.

Response data fields

    Example

    curl "https://api.gridlode.com/v1/url/crawl?url=https%3A%2F%2Fexample.com%2F&limit=2" \
  -H "Authorization: Bearer $GRIDLODE_KEY"

    Try it without code on the Website Crawler (Markdown for RAG) API page.