How Blackreach works, and why I built it differently

Every autonomous web agent I tried had the same problem. It worked on the demo site and fell apart on anything real. Cloudflare would catch it in seconds. JavaScript-rendered content was completely invisible to it. Rate-limit responses came back as 200 OK with an error page in the body, and the agent would report success, save garbage, and move on. I'd find out hours later when I opened the file.

The specific task that pushed me to build this was downloading the full Linear A inscription corpus from sigla.phis.me. 847 inscriptions, each with photographs, sign annotations, and transliterations. No public API. Data scattered across hundreds of pages with JavaScript-rendered pagination. I tried three frameworks. All of them died on page two. One saved 847 copies of the same 403 error page and reported 100% success. That's when I stopped looking for an existing solution.

The ReAct loop

At the core is a ReAct loop. The agent gets a task, reasons through what to do next, takes an action, observes what happened, and repeats. The loop isn't the interesting part. The observation is.

Thought: I need the inscription table on this page
Action: navigate("https://sigla.phis.me/")
Observation: Page loaded. Nav: [About, Database, Signs].
  Main: table, 847 rows, columns [ID, Site, Text, Image].
  Interactive: pagination controls, export button.
Thought: extract all rows and handle pagination
Action: extract_table(selector=".inscription-table", paginate=True)

Most agents just dump raw HTML into the context. A typical page runs 50k to 500k tokens of noise and the model gets buried in it. Blackreach uses a DOM walker instead. It pulls out the semantic structure: visible text, interactive elements, nav landmarks, ARIA roles. A 200k-token page becomes a 2k-token observation the LLM can actually work with.

The DOM walker also assigns numeric IDs to every interactive element. The model clicks [15], not a CSS selector it has to guess or a brittle XPath. If the page structure changes between visits, which it often does, the IDs update automatically. The model doesn't need to know CSS. It just needs to know what's on the page right now.

Stealth Playwright

Standard Playwright gets caught immediately. The tells are well documented: navigator.webdriver = true, missing browser extensions, CDP artifacts, headless viewport signatures, unnatural input timing. Any halfway-serious anti-bot system checks for all of these at once.

But it goes deeper than the obvious ones. Real browsers have canvas fingerprints. They have WebGL renderer strings that match their reported GPU. Mouse movements follow curves with natural variance, not perfectly straight lines. Keystroke timing varies the way human typing varies. Blackreach patches these at the browser level before any page loads: bezier mouse trajectories with jitter, keystroke timing pulled from a distribution of real typing speeds, viewport dimensions from a lookup table of actual common screen sizes, WebGL strings that match the user agent.

Session resume and memory

Research tasks take time. A full database download across hundreds of paginated pages can run for hours, and interruptions happen. Blackreach auto-saves task state when interrupted. Every downloaded file, visited URL, and decision point gets written to SQLite as it happens. When you resume, it picks up exactly where it left off and doesn't fetch the same file twice.

The cross-session memory goes further. Blackreach remembers what worked per domain. If a particular site needs a three-second wait before pagination renders, that gets stored. Next session, it already knows. It gets better at the sites it visits repeatedly.

Why so many tests

Autonomous agents fail silently. That's the thing that kept breaking my trust in them. The agent says it succeeded, the file is there, and you don't find out until you open it and it's a 403 page saved as HTML.

Every test in the suite came from a real failure. A rate limit returns 200 OK with a JSON body that says "success": true but an empty data field. A pagination button exists in the DOM but inside a hidden div that only becomes visible after a 1.5-second delay. Navigate too fast and you're clicking nothing. A login wall doesn't appear on the first ten pages; it triggers on page eleven based on session age, and only from non-residential IPs. These aren't edge cases I invented to pad a count. They're sites I actually tried to scrape. When it's running at 3am collecting data, I need it to fail loud.

Open source on github and gitlab. Full feature breakdown on the project page.