# Spider Crawler

High-performance Rust web crawler & scraper generating LLM-ready markdown

Spider is an ultra-fast web crawler written in Rust that streams clean Markdown, structured JSON, and embeddings for AI models.

## Overview

- **Website**: https://spider.cloud
- **Docs**: https://spider.cloud/docs
- **GitHub**: https://github.com/spider-rs/spider
- **License / Pricing**: Open Source
- **Directory Entry**: https://agentaccess.dk/enablers/spider

## About

Spider is engineered for maximum crawling throughput. Written in Rust, it defaults to lightweight HTTP requests for static content and automatically escalates to headless browser rendering only when JavaScript is strictly required, generating pristine LLM markdown at extreme speed.

## Key Features

- Concurrent multi-threaded crawling in Rust
- Smart mode: runs HTTP for static pages, headless browser for JS
- Built-in AI extraction with natural language prompt filters
- Official SDKs for Rust, Python, TypeScript, and Node.js

## Danish System Application Recipes

### Crawl Entire Danish Public Sector Portals in Seconds
Run Spider to crawl and convert large Danish government guidance directories (e.g. Borger.dk or Skat.dk) into indexed Markdown collections.

## Quickstart

```typescript
import { Spider } from '@spider-rs/spider-client'

const app = new Spider({ apiKey: process.env.SPIDER_API_KEY })
const result = await app.scrapeUrl('https://www.borger.dk', {
  return_format: 'markdown',
})

console.log(result[0].content)
```

---
Source: [AgentAccess Enablers](https://agentaccess.dk/enablers/spider)