Hybrid fetch mode
Uses a lightweight static HTTP fetch first, then falls back to Playwright only when rendering or bot protection appears necessary.
LightCrawl
LightCrawl is a self-hostable web scraping API and MCP server that converts web pages into clean Markdown for local tools and developer workflows. It supports HTTP access, MCP clients, article-focused extraction, full-page conversion, and optional Redis-backed crawling.
Overview
LightCrawl is a self-hostable web scraping API and Model Context Protocol (MCP) server that converts web pages into clean Markdown. It is designed as a lightweight, single-container alternative for developers who want page extraction, site mapping, and crawling without running a full browser stack for every request.
The project emphasizes low-resource operation and flexible deployment. According to the README, it can serve HTTP requests through an Express API, answer MCP clients over stdio, and switch between static fetching and browser-based rendering depending on the page.
The repository documentation also positions LightCrawl for local development, Docker-based deployment, and Redis-backed distributed crawling when multiple workers or queued jobs are needed. Its public pages describe it as open source and self-hostable rather than a hosted SaaS product.
Core capabilities
Fast response path
Supports a `fast=true` mode that skips the scrolling delay and uses a lighter DOM parser to reduce CPU and memory use.
Dynamic content handling
Automatically scrolls pages and captures rendered HTML when browser fallback is needed, which helps with lazy-loaded content.
Two extraction modes
Converts pages to Markdown with a Readability-based `article` mode and a `full` mode for complete HTML-to-Markdown conversion.
Crawl and map workflow
Finds internal URLs for mapping, supports recursive crawling, and can run the crawl queue in Redis for distributed processing.
Operational and client interfaces
Provides Prometheus-compatible metrics, structured JSON logs, and an MCP server interface alongside the HTTP API.
Common use cases
Convert articles to readable text
Use the `article` mode when you want to pull the main body of blog posts, news articles, or documentation pages into clean Markdown for downstream processing.
Capture complete page content
Use the `full` mode when you need the entire page body, such as directory pages, portals, or search results, instead of filtering to the main article content.
Map or crawl a website
Use the mapping and crawl endpoints to discover internal links from a target domain and then recurse through a site up to configured depth and count limits.
Self-host in an isolated environment
Run the service in Docker or as a single self-hosted container when you want an isolated scraper that keeps browser execution off the client machine.
Scale crawl workloads with Redis
Enable Redis-backed crawling when a project needs shared queue state or cooperative background workers across multiple jobs.
Pros and Cons
Pros
- Combines scraping, mapping, and crawling in one self-hostable service.
- Uses a lightweight static-first flow that can avoid browser rendering when a page is simple.
- Provides an article-focused extraction mode for cleaner content and a full mode for broader page conversion.
- Supports Redis-backed distributed crawling for larger workloads.
- Includes observability features such as Prometheus-compatible metrics and structured logs.
Cons
- The source material is limited on client/library compatibility beyond HTTP API and MCP server support.
- Some operational details, such as exact output schemas and advanced integrations, are only partially documented in the provided sources.
FAQ
How is LightCrawl deployed and accessed?
LightCrawl runs as a self-hosted service. The README says it works as both an Express-based HTTP API and an MCP server, with configurable access controls such as an optional API key and allowed IP list.
What does LightCrawl extract from a web page?
The README documents two extraction modes: `article` for primary article content and `full` for converting the entire HTML body to Markdown. It also includes endpoints for scraping, mapping, and crawling.
Does LightCrawl support distributed crawling?
The project supports both in-memory queues and Redis-backed queues for crawling. When `REDIS_URL` is set, it can use Redis as a distributed queue and state backend, with an optional background worker enabled by `ENABLE_DISTRIBUTED_WORKER`.
Can LightCrawl run in Docker?
The README lists a Dockerfile, a single-container setup, and a multistage build optimized for Playwright with Chromium only, which makes it suitable for container-based self-hosting.
Is LightCrawl itself a paid service?
The public GitHub pricing page shows a Free plan and paid plans on GitHub itself. That indicates the project is hosted on GitHub, but the repository README still describes LightCrawl as a self-hostable open-source project rather than a hosted SaaS product.
Quick Facts
- Category
- Developer Tool
- Product type
- Self-hostable web scraping API and MCP server
- Primary output
- Web pages converted to Markdown
- Deployment
- Single-container, Docker-friendly
- Runtime
- Node.js 18+
- Source domain
- github.com
LightCrawl Alternativen
ByteAsk
ByteAsk is a terminal-first AI coding agent for C and C++ that edits repositories and verifies changes with the real compiler, debugger, sanitizers, and tests before showing a diff. It offers a free tier plus paid plans, with editor connectors and zero-retention handling described in the source.
CreateOS Sandbox
CreateOS Sandbox is an isolated compute environment for running code and agent workloads inside Firecracker micro-VMs. It is designed for workflows that need machine-level isolation, private networking between sandboxes, and programmatic control through SDK, CLI, or MCP.
hob
hob is an independent workspace for coding agents that keeps agent sessions, terminals, history, and follow-up work organized around the tools and providers you already use. It is aimed at developers who want local control over routing, history, and workspace structure rather than a bundled model stack.
Manta AI
Manta AI is an autonomous web app testing tool for teams that want to map application behavior, catch regressions, and generate tests without writing scripts or maintaining selectors. It works from a URL and supports plain-English test flows, run results with screenshots, and scheduled or deployment-triggered checks.
SonOf
SonOf connects to your repo and PM tool, audits the codebase and surrounding product context, and turns approved work into shipped tickets with senior engineering review. It is aimed at founders and engineering leaders who need backlog help without hiring a full team immediately.
Ghost
Ghost ist ein terminalbasierter KI-Assistent für Chats, Code-Generierung und Aufgaben direkt in der Kommandozeile. Mit kostenlosen Modellen, Linux, macOS und Windows. Open Source.