Spider Crawler
High-performance Rust web crawler & scraper generating LLM-ready markdown
Spider is an ultra-fast web crawler written in Rust that streams clean Markdown, structured JSON, and embeddings for AI models.
Kom hurtigt i gang
crawl.ts
Kom i gang med Spider Crawler i dit projekt.
import { Spider } from '@spider-rs/spider-client'
const app = new Spider({ apiKey: process.env.SPIDER_API_KEY })
const result = await app.scrapeUrl('https://www.borger.dk', {
return_format: 'markdown',
})
console.log(result[0].content)Om Spider Crawler
Spider is engineered for maximum crawling throughput. Written in Rust, it defaults to lightweight HTTP requests for static content and automatically escalates to headless browser rendering only when JavaScript is strictly required, generating pristine LLM markdown at extreme speed.
Danske data- og produkt-guides
Hvordan udviklere og agenter bruger Spider Crawler til at tilgå dansk digital infrastruktur.
Crawl Entire Danish Public Sector Portals in Seconds
Run Spider to crawl and convert large Danish government guidance directories (e.g. Borger.dk or Skat.dk) into indexed Markdown collections.