Spider Crawler
High-performance Rust web crawler & scraper generating LLM-ready markdown
Spider is an ultra-fast web crawler written in Rust that streams clean Markdown, structured JSON, and embeddings for AI models.
Quickstart Execution
crawl.ts
Get started with Spider Crawler in your project.
import { Spider } from '@spider-rs/spider-client'
const app = new Spider({ apiKey: process.env.SPIDER_API_KEY })
const result = await app.scrapeUrl('https://www.borger.dk', {
return_format: 'markdown',
})
console.log(result[0].content)About Spider Crawler
Spider is engineered for maximum crawling throughput. Written in Rust, it defaults to lightweight HTTP requests for static content and automatically escalates to headless browser rendering only when JavaScript is strictly required, generating pristine LLM markdown at extreme speed.
Danish Product & Data Recipes
How developers and agents use Spider Crawler to connect with Danish digital infrastructure.
Crawl Entire Danish Public Sector Portals in Seconds
Run Spider to crawl and convert large Danish government guidance directories (e.g. Borger.dk or Skat.dk) into indexed Markdown collections.