Blackreach · autonomous browser agent

live demo loop · agent running a real taskfull 4 min run below ↓

blackreach run "download all Linear A inscription tables from sigla.phis.me"

Why it exists

Every autonomous web agent I tried had the same problem. It worked on the demo site and fell apart on anything real. Cloudflare caught it in seconds. JavaScript-rendered content was invisible to it. Rate limit responses came back as 200 OK with an error page in the body, and the agent saved garbage, reported success, and moved on. I found out hours later when I opened the file.

The task that broke me was downloading the full Linear A inscription corpus. 847 inscriptions, no public API, JavaScript pagination. Three frameworks died on page two. One of them saved 847 copies of the same 403 page and reported 100% success. So I stopped looking and built my own.

How the loop works

TASK "download all Linear A inscriptions" │ ▼ OBSERVE ── DOM walker strips the page to what matters │ raw page: 50k-500k tokens of html noise │ observation: ~2k tokens · text, links, landmarks, │ every interactive element tagged with an id [15] ▼ THINK ──── llm reasons over the observation, picks an action │ ollama / openai / anthropic / google / xai ▼ ACT ────── stealth playwright executes it │ click [15] · extract_table · navigate · download ▼ verify → log to sqlite → loop until done │ ├─ stuck? loop detection switches strategy / fails over └─ interrupted? state saved, resume picks up exactly here

The loop is standard ReAct. The observation is the part that matters. Most agents dump raw HTML into context and the model drowns. The DOM walker pulls out visible text, interactive elements, nav landmarks and ARIA roles, then numbers every clickable thing. The model clicks [15], not some CSS selector it has to guess. Page structure changes between visits, the ids update, nothing breaks.

Stealth that actually holds up

Standard Playwright gets caught instantly. The tells are documented everywhere: navigator.webdriver, missing plugins, CDP artifacts, headless viewport sizes, robotic input timing. Any serious anti-bot system checks all of them at once.

Blackreach patches the browser before any page loads. Mouse moves on bezier curves with jitter. Keystroke timing pulls from a distribution of real typing speeds. Viewport sizes come from a table of actual common screens. WebGL strings match the reported GPU. The automation fingerprint is gone before the first request goes out.

Not undetectable. Nothing is. It passes basic Cloudflare, most Akamai setups, and the rate limiters on academic databases. That covers the research work I built it for.

Built for long runs

session resume

Every file, URL and decision is written to SQLite as it happens. Interrupt it, come back, it continues from the exact step.

cross-session memory

Remembers what worked per domain. If a site needs a 3 second wait before pagination renders, next session it already knows.

smart dedup

URL plus content hash. It never downloads the same file twice, even across sessions.

stuck detection

Loop detection with automatic strategy switching and source failover when a site stonewalls.

MCP server

Claude Desktop integration for JS-rendered and anti-bot pages.

REST API

FastAPI server with an async job queue for programmatic runs.

Why 2,973 tests

Autonomous agents fail silently. That is the thing that kept breaking my trust. The agent says it succeeded, the file exists, and you find out later it is a 403 page saved as HTML.

Every test in the suite came from a real failure. A rate limit that returns 200 OK with "success": true and an empty data field. A pagination button that exists in the DOM but sits in a hidden div for 1.5 seconds after load, so clicking too fast clicks nothing. A login wall that only triggers on page eleven, based on session age, and only from non-residential IPs. None of these are invented to pad a number. They are sites I actually tried to scrape. When this thing runs at 3am collecting data, it has to fail loud.

Watch a real run

full demo · unedited 4 min run1280p · 4.3MB

Open source on github and gitlab. Longer write-up: how Blackreach works →