Live
URL to Markdown
The main content of a web page as clean Markdown (boilerplate, navigation and ads removed) with title, description, author, publish date and language
Web pages and crawling · 13 APIs
Web page and crawling APIs
Any public page as HTML (with JavaScript rendering), Markdown for LLMs, meta tags, links or a screenshot. Crawl whole sites, find broken links and measure Core Web Vitals.
Single-page engines cost 1 to 3 credits per URL, and the crawler and broken-link checker bill per page crawled. Screenshots and the speed test run in real Chrome, the crawler can render JavaScript too, and the audits (meta tags, security headers, AI readiness) return a score with plain-English fixes. Every engine respects robots.txt by default and failed fetches are not charged.
- APIs
- 13
- Live now
- 13 of 13
- From
- 1 credit / request
- Free tools
- 12
Start here
Most-used APIs in Web pages and crawling
Live
Website Crawler (Markdown for RAG)
Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word…
1 credit / pageDocs
Live
Screenshot
A screenshot of any public web page, rendered in real Chrome with JavaScript
All 13 APIs in Web pages and crawling
13 live · 0 launching soonURL to Markdown API
The main content of a web page as clean Markdown (boilerplate, navigation and ads removed) with title, description, author, publish date and language
1 credit / requestLiveWebsite Crawler (Markdown for RAG) API
Crawl a website from a start URL (and optionally its XML sitemap) and get every page back as clean Markdown with title, description, language, word…
1 credit / pageLiveScreenshot API
A screenshot of any public web page, rendered in real Chrome with JavaScript
2 credits / requestLivePage HTML API
The raw HTML of any public web page with final URL, status code, redirect chain and key headers
1 credit / requestLivePageSpeed & Core Web Vitals Test API
Load a page in real headless Chrome (mobile or desktop emulation, as in Lighthouse) and measure it
3 credits / requestLiveBroken Link Checker API
Crawl a website and check every internal and external link for 404s, server errors, timeouts and redirect chains
1 credit / pageLiveArticle Extractor & Readability API
Reader-mode extraction: article text as Markdown with author, dates, reading time and a readability score
1 credit / requestLiveGEO / AI-Readiness Audit API
How ready a page is to be read, understood and cited by AI search engines and assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot)
2 credits / requestLiveHTTP Headers & Security Check API
The HTTP response headers of any URL with the full redirect chain (status and timing of every hop), compression, caching and CDN details, and a…
1 credit / requestLiveKeyword Density API
The words and phrases (1-3 word n-grams) a page repeats most, with counts and density as a percentage of all words
1 credit / requestLivePage Links Extractor API
Every link on a web page as an absolute URL with its anchor text, internal/external type and rel flags (nofollow, sponsored, ugc), plus totals and…
1 credit / requestLivePage Metadata & SEO Tags API
Every on-page SEO signal of a web page in one call
1 credit / requestLiveSitemap Parser API
Every page URL listed in a website's XML sitemaps, with last-modified date, change frequency and priority
2 credits / requestLive
Solutions
What people build with these APIs
- SEO and rank trackingRankings, backlinks, authority and site auditsUses 7 APIs from this category
- AI agents and RAGGive LLMs live web data and clean MarkdownUses 4 APIs from this category
- Competitor and market researchSee a competitor's stack, traffic signals and changesUses 2 APIs from this category
- E-commerce and price monitoringTrack products, prices and stockUses 1 API from this category
- Security and domain monitoringExpiry, SSL, DNS and email-security alertsUses 1 API from this category
Free tools
Try them without an account
The same engines, in your browser. No sign-up.
FAQ
Web pages and crawling: common questions
Every API also comes with failed requests free, half-price cached results and credits that never expire.
See pricingWhich output should I feed to an LLM?
URL to Markdown for one page, the website crawler for a whole site, or the article extractor for news and blog posts. All three strip navigation, ads and boilerplate, so your tokens go to the content.
Do you respect robots.txt?
Yes, by default. For sites you own or have permission to process, the API lets you turn respect_robots off.
How is crawling billed?
The website crawler costs 1 credit per page returned, or 2 with JavaScript rendering. The broken-link checker costs 1 credit per page crawled, and the link checks themselves are free. Failed crawls cost nothing.
More categories
Call any of these 13 APIs with one key.
1,000 free credits when you sign up, no card required. Failed requests are never charged.