Website Crawler (Markdown for RAG)
Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word…
The main content of a web page as clean Markdown (boilerplate, navigation and ads removed) with title, description, author, publish date and language. Built for LLMs and RAG pipelines. Respects robots.txt by default.
Try . Free, no account needed.
| Name | Type | Description |
|---|---|---|
urlrequired | string | Public http(s) URL to convert. |
include_links | boolean | Keep hyperlinks in the Markdown. Default: true |
respect_robots | boolean | Refuse URLs disallowed by the site's robots.txt. Default: true |
max_age | integer | Accept a cached answer up to this many seconds old, billed at 50%. Default 0 (always fresh). |
title / heading · The HTML <title> and the main content heading.description / author / published / site_name · Metadata detected on the page.language · Language from the html lang attribute.markdown · Main content as Markdown, without navigation, ads or footers.word_count · Words in the Markdown.curl "https://api.gridlode.com/v1/url/markdown?url=https%3A%2F%2Fexample.com%2F" \
-H "Authorization: Bearer $GRIDLODE_KEY"Market reference: TabAPI Web Reader: 1 credit ($1.00/1k) · Firecrawl $0.6-3.8/1k.
Yes. Boilerplate is removed so you send only the main content, which cuts tokens and improves answer quality.
1 credit per request. One credit costs $0.001 on the smallest pack and as little as $0.0003 on enterprise plans. Cached answers cost half and failed calls are free.
Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word…
A screenshot of any public web page, rendered in real Chrome with JavaScript
The raw HTML of any public web page with final URL, status code, redirect chain and key headers
Load a page in real headless Chrome (mobile or desktop emulation, as in Lighthouse) and measure it
Crawl a website and check every internal and external link for 404s, server errors, timeouts and redirect chains
Reader-mode extraction: article text as Markdown with author, dates, reading time and a readability score