Hybrid fetch mode
Uses a lightweight static HTTP fetch first, then falls back to Playwright only when rendering or bot protection appears necessary.
LightCrawl
LightCrawl is a self-hostable web scraping API and MCP server that turns web pages into clean Markdown for local tools and developer workflows.
Overview
LightCrawl is a self-hostable web scraping API and Model Context Protocol (MCP) server that converts web pages into clean Markdown. It is designed as a lightweight, single-container alternative for developers who want page extraction, site mapping, and crawling without running a full browser stack for every request.
The project emphasizes low-resource operation and flexible deployment. According to the README, it can serve HTTP requests through an Express API, answer MCP clients over stdio, and switch between static fetching and browser-based rendering depending on the page.
The repository documentation also positions LightCrawl for local development, Docker-based deployment, and Redis-backed distributed crawling when multiple workers or queued jobs are needed. Its public pages describe it as open source and self-hostable rather than a hosted SaaS product.
Core capabilities
Fast response path
Supports a `fast=true` mode that skips the scrolling delay and uses a lighter DOM parser to reduce CPU and memory use.
Dynamic content handling
Automatically scrolls pages and captures rendered HTML when browser fallback is needed, which helps with lazy-loaded content.
Two extraction modes
Converts pages to Markdown with a Readability-based `article` mode and a `full` mode for complete HTML-to-Markdown conversion.
Crawl and map workflow
Finds internal URLs for mapping, supports recursive crawling, and can run the crawl queue in Redis for distributed processing.
Operational and client interfaces
Provides Prometheus-compatible metrics, structured JSON logs, and an MCP server interface alongside the HTTP API.
Common use cases
Convert articles to readable text
Use the `article` mode when you want to pull the main body of blog posts, news articles, or documentation pages into clean Markdown for downstream processing.
Capture complete page content
Use the `full` mode when you need the entire page body, such as directory pages, portals, or search results, instead of filtering to the main article content.
Map or crawl a website
Use the mapping and crawl endpoints to discover internal links from a target domain and then recurse through a site up to configured depth and count limits.
Self-host in an isolated environment
Run the service in Docker or as a single self-hosted container when you want an isolated scraper that keeps browser execution off the client machine.
Scale crawl workloads with Redis
Enable Redis-backed crawling when a project needs shared queue state or cooperative background workers across multiple jobs.
Pros and Cons
Pros
- Combines scraping, mapping, and crawling in one self-hostable service.
- Uses a lightweight static-first flow that can avoid browser rendering when a page is simple.
- Provides an article-focused extraction mode for cleaner content and a full mode for broader page conversion.
- Supports Redis-backed distributed crawling for larger workloads.
- Includes observability features such as Prometheus-compatible metrics and structured logs.
Cons
- The source material is limited on client/library compatibility beyond HTTP API and MCP server support.
- Some operational details, such as exact output schemas and advanced integrations, are only partially documented in the provided sources.
FAQ
How is LightCrawl deployed and accessed?
LightCrawl runs as a self-hosted service. The README says it works as both an Express-based HTTP API and an MCP server, with configurable access controls such as an optional API key and allowed IP list.
What does LightCrawl extract from a web page?
The README documents two extraction modes: `article` for primary article content and `full` for converting the entire HTML body to Markdown. It also includes endpoints for scraping, mapping, and crawling.
Does LightCrawl support distributed crawling?
The project supports both in-memory queues and Redis-backed queues for crawling. When `REDIS_URL` is set, it can use Redis as a distributed queue and state backend, with an optional background worker enabled by `ENABLE_DISTRIBUTED_WORKER`.
Can LightCrawl run in Docker?
The README lists a Dockerfile, a single-container setup, and a multistage build optimized for Playwright with Chromium only, which makes it suitable for container-based self-hosting.
Is LightCrawl itself a paid service?
The public GitHub pricing page shows a Free plan and paid plans on GitHub itself. That indicates the project is hosted on GitHub, but the repository README still describes LightCrawl as a self-hostable open-source project rather than a hosted SaaS product.
Quick Facts
- Category
- Developer Tool
- Product type
- Self-hostable web scraping API and MCP server
- Primary output
- Web pages converted to Markdown
- Deployment
- Single-container, Docker-friendly
- Runtime
- Node.js 18+
- Source domain
- github.com
LightCrawl Alternatives
ByteAsk
ByteAsk is a terminal-first AI coding agent for C and C++ that edits repos and verifies changes with compilers, debuggers, sanitizers, and tests.
CreateOS Sandbox
CreateOS Sandbox is an isolated compute environment for running code and agent workloads in Firecracker micro-VMs with private networking and SDK, CLI, or MCP control.
hob
hob is an independent workspace for coding agents, with local control over sessions, terminals, history, routing, and follow-up work.
Manta AI
Manta AI is an autonomous web app testing tool that maps app behavior, catches regressions, and generates tests from a URL, no scripts or selectors needed.
SonOf
SonOf connects to your repo and PM tool, audits your codebase, and turns approved work into shipped tickets with senior engineering review.
Ghost
Ghost is a terminal-based AI assistant for chatting, code generation, and CLI tasks. Includes free models, supports Linux, macOS, Windows, and is open source.