Crawl4AI
Open-source, blazing fast asynchronous LLM web crawler
Crawl4AI is an open-source, speed-optimized asynchronous web crawler that generates structured Markdown and JSON graphs for AI models.
Quickstart Execution
crawler.py
Get started with Crawl4AI in your project.
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://www.retsinformation.dk")
print(result.markdown[:500])
asyncio.run(main())About Crawl4AI
Crawl4AI is engineered specifically for AI engineers who want zero subscription fees and extreme throughput. It runs asynchronous crawls with smart caching, CSS-selector stripping, heuristic content scoring, and multi-URL batching, producing clean LLM markdown with metadata at zero cost.
DK
Danish Product & Data Recipes
How developers and agents use Crawl4AI to connect with Danish digital infrastructure.
Batch Ingest Danish Legal Codes from Retsinformation.dk
Run Crawl4AI across Danish legal statues and minister orders on Retsinformation to generate an open-source vector store of Danish law.
Key Capabilities
100% open source and local execution
Asynchronous batch crawling with asyncio
Heuristic content extraction (removes headers, footers, and noise)
Extracts structured JSON using local models (Ollama) or cloud APIs
Screenshots and PDF extraction capabilities