Skip to content
Gridlode
LiveWeb pages and crawling1 credit / page

Website Crawler (Markdown for RAG) API

Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word count and link depth. Built for RAG and LLM ingestion: navigation and boilerplate removed, duplicates dropped, very long pages truncated and flagged. Same-site only, include/exclude path patterns (glob or regex), optional JavaScript rendering in real Chrome. Polite by design: robots.txt and Crawl-delay respected, at most 4 requests in flight, hard page, byte and time limits. Charged per page returned (1 credit, 2 with rendering); failed crawls are free.

GEThttps://api.gridlode.com/v1/url/crawl
Get an API keyReference docs

Parameters

NameTypeDescription
urlrequiredstringStart URL: a site root or any page. Only pages on the same site are crawled (same registrable domain, so www. and other subdomains are included).
limitintegerMaximum number of pages to return. Charged per page returned: 1 credit, 2 with render=true.
Default: 10
max_depthintegerHow many links deep to follow from the start page (0 = only the start page and, with use_sitemap, the sitemap URLs).
Default: 2
include_pathsarray of stringOnly crawl URLs whose path matches one of these patterns. Glob (/blog/*, *.html), regex (prefix re: or start with ^), or a plain path prefix (/docs). Matched against the URL path and query. The start page is always read for links.
exclude_pathsarray of stringNever crawl URLs whose path matches one of these patterns (exclude wins over include). Glob (/blog/*, *.html), regex (prefix re: or start with ^), or a plain path prefix (/docs). Matched against the URL path and query.
use_sitemapbooleanAlso seed the crawl with the URLs listed in the site's XML sitemap (found through robots.txt or /sitemap.xml).
Default: false
renderbooleanRender every page in real Chrome so JavaScript-built content is captured. Slower; 2 credits per page.
Default: false
include_linksbooleanKeep hyperlinks in the Markdown.
Default: false
respect_robotsbooleanSkip URLs disallowed by the site's robots.txt and honour its Crawl-delay.
Default: true
max_ageintegerAccept a cached answer up to this many seconds old, billed at 50%. Default 0 (always fresh).

Response fields

    Code

    curl "https://api.gridlode.com/v1/url/crawl?url=https%3A%2F%2Fexample.com%2F&limit=2" \
  -H "Authorization: Bearer $GRIDLODE_KEY"
    1 credit per page
    One credit is $0.001 on the smallest pack, from $0.0003 on enterprise plans.
    Half price for cached data
    Cached for up to 1 h. Pass max_age to use it.
    Failed requests are free
    Blocks, timeouts and target errors never cost credits.

    Market reference: Firecrawl crawl: 1 credit/page ($0.83-5.33/1k) · Apify Website Content Crawler ~$0.2-5/1k pages · Spider.cloud ~$0.5-1/1k.

    Questions

    Part of these solutions
    How much does the Website Crawler (Markdown for RAG) API cost?

    1 credit per page. One credit costs $0.001 on the smallest pack and as little as $0.0003 on enterprise plans. Cached answers cost half and failed calls are free.

    Web pages and crawling

    Related APIs in this category

    All 13 in Web pages and crawling · 13 live
    Live

    URL to Markdown

    The main content of a web page as clean Markdown (boilerplate, navigation and ads removed) with title, description, author, publish date and language

    1 credit / requestTry freeDocs
    Live

    Screenshot

    A screenshot of any public web page, rendered in real Chrome with JavaScript

    2 credits / requestTry freeDocs
    Live

    Page HTML

    The raw HTML of any public web page with final URL, status code, redirect chain and key headers

    1 credit / requestTry freeDocs
    Live

    Broken Link Checker

    Crawl a website and check every internal and external link for 404s, server errors, timeouts and redirect chains

    1 credit / pageTry freeDocs