Optimizing the browser agent's tools, from its own JavaScript history

SECTION 1

Investigation

1.1  Scraping the history

The entire history of Polar's JS calls, split simply by domain.

1,657,377 calls. 82,196 runs. 5,456 users. 44,636 sites. This is all of it: the first JavaScript call is dated 8 July 2026. These 20 sites hold 47.5% of every call. The grey rows are a handful of people's bespoke automation. Section 1.2 deals with them.

Highlights of the work

• Built a custom, resumable pipeline over the whole production trace database. It rebuilds the JavaScript the agent ran, call by call.
• Found the exact source of a call even when the trace did not hold it. The agent writes a script to a file, edits it, then runs it from disk. So I replay the tool calls and simulate a filesystem. That recovers 119,267 otherwise invisible calls. Exact code now covers 1,349,302 of 1,657,377 calls.
• Found the exact URL for 99.93% of calls. 228,214 of them carry no URL at all. A per-tab URL timeline rebuilds those. Without this there is no per-site work.
• Reconciled the result instead of assuming it. All but 0.07% of runs match Postgres. 42 tests, each one mutation-checked.

1.2  What is the actual head of distribution?

The GitHub gist I was given named a few head-of-distribution sites. But it called its own method a weak proxy. It only counted how often an https string appeared in traces. So I could not use it to find the real head of distribution.

Many other problems made this a data engineering task as much as an ML tooling task.

• Some sites have a huge call count from one user's bespoke automation. thedailyvirals.com has 11,340 calls from 1 user.
• On many sites the calls are very short. Those are not worth replacing with a tool.
• Some sites have both volume and spread, and yet one user still dominates. That is a positive and a negative signal at the same time. Google has 624 users, and one of them makes 35.8% of its calls.

What failed

• Sort by a primary column, then break ties on a second. The primary column decides everything.
• A weighted score, α·calls + β·users. The two columns differ by two orders of magnitude. The constant becomes the answer.
• A threshold on calls, then a sort by users. This moves the arbitrary number. It does not remove it.
• Rank on raw user count. That counts heads only. A user with one call weighs as much as one with half the traffic.

What worked

• The Herfindahl–Hirschman index, inverted. It measures spread, not volume. Effective users is 1 / Σ(share²) over each user's share of a site's calls. It is the number of users the traffic behaves like. LinkedIn's top user makes 4.6% of its calls, and its 1,309 users behave like 110. Google's top user makes 35.8%, and its 624 users behave like 6.7.
• Volume still counts. So the HHI rank is added to the call rank, and a site must rank high on both. Ties go to the higher call count.
• A floor of 50 users per site, applied before the ranking.
• Merged white-labeled sites. Each tenant arrives as its own subdomain of one product. Instructure shot up. Alone, portland.instructure.com is 555 calls from 53 users, at rank 23. Merged, *.instructure.com is 3,068 calls from 141 users, at rank 17. Shopify, SharePoint, Atlassian and Workday merge the same way.

The true head of distribution. Top 25 sites.

Every single-user host from the first table is gone. Instagram falls from 2nd to 16th. Google falls from 3rd to 18th. One user drives about a third of each. docs.google.com is 23rd by call count alone, and 11th here. Instructure's Canvas reaches 17th once its schools merge.

1.3  Extracting the signal to automate: the crux

Now I know which sites to focus on. Next I must find what actions get expensively rewritten in JavaScript, over and over.

This is hard. The same macro-action can be written a hundred different ways. So a capability is not a byte-identical script. To find what is worth compressing into a tool, I have to group by capability. Byte-identical groups are a healthy fraction, but a very long tail sits behind them.

What I tried

• AST parsing.
• MinHash and code embeddings, to extract the semantics of the code.
• The agent's own code description. No information: 139,090 distinct captions for 228,101 calls.

None of them worked. The best covers only 34% of LinkedIn's traffic with its top 25 groups. The table scores them against two baselines: the raw script, and the script with its literals normalized. The number is that share, measured on LinkedIn.

So I used a special workflow

1. Clean. Remove the comments. Lint every script into the same form. Drop the scalar literals. That string is the crux.
2. Drop the trivia. A crux over 200 characters always counts. Below that I drop bare property reads, bare calls, and anything under 60 characters that projects nothing. That removes 102,555 calls.
3. Group on the crux. Identical, or 95% similar.
4. Rank on how many times that same script was called across runs.
5. Extract, 5× per site. Sample 150 scripts at random, weighted toward higher rank and popularity. Give them to a Haiku 5.5 at default effort. Ask it for every capability, as a list of parametrized JS scripts. Tell it to favor low latency and token-efficient returns. Do this for all 25 sites.
6. Merge. Opus agents and fresh Haikus take the 125 reports. They extract the JS functions common within one site, then across sites.

That is 125 claude -p runs. They read 18,750 scripts and proposed 1,434 capabilities with code. Every run went in parallel. It took 14 minutes, and only 10% of a subscription's five-hour window. In series it would have taken 27 hours.

SECTION 2

Delivered to the harness

2.1  High-signal new general tools, and improvements to existing ones

Six changes. Three new tools, and three improvements to tools that already exist. All six are built, registered and verified in Polar's own Chromium.

What pain does this help with? Every frequency below covers all runs, not just the head of distribution.

Measured over 1,648,945 calls, 80,916 runs, 5,409 users and 44,635 sites. These are the calls made on a site; the other 8,432 ran on no page. A match needs the recovered script text. That exists for 81% of calls. So every count here is a lower bound.

Fails today is the share of those calls that do not return ok. The corpus baseline is 6.2%. The empty-result row is the exception. Those all fail by definition, so the rate says nothing.

2.2  High-signal new site-specific toolkits

The six changes above supply the scrolling, the link scan and the form read. They do not supply the meaning. The code that turns one card or one row into named fields shares no line between sites. A site toolkit supplies that.

Built from the 25 site reports in section 1.3: 26 toolkits, 100 verified readers, each one derived from the production scripts for that page. The agent calls them with execute_javascript(toolkit="<site>") instead of writing DOM code.

2.3  Making breakage loud: drift detection and self-healing

A site changes its markup and a hand-written selector returns []. Nothing errors, so the agent treats an empty page as an empty result. Every reader therefore marks its own outcome against three anchors.

• surface — the container that proves this is the right page. Missing, and the status is drift.
• data — the row nodes. One or more, and the status is ok.
• empty — the site's own empty-state node. Zero rows with that node present is empty; zero rows without it is drift.

So zero rows only counts as empty when the page itself says it is empty. Anything else is reported as breakage. Every call returns the verdict alongside the data:

The tool wraps every exported function before the agent's code runs, so the verdict is recorded where the call happens and the agent cannot discard it. On drift the tool hands back the toolkit's own source, numbered, with an instruction to repair it:

This ran for real. In one Instagram lead-generation run the profile reader drifted 98 times, the agent read the page, and made a two-character widening:

After the edit: 0 drifts and 35 successful calls for the rest of the run. The repair lives in the sandbox for that run only, so it never silently becomes the shipped toolkit.

SECTION 3

A/B testing

I took real user prompts out of the production trace database and replayed each one twice against the same live sites: once on main, once on the branch carrying the six tool changes and the 26 site toolkits. Everything else is the same commit, so the tooling is the only difference, and every figure below comes from Polar's own tables.

Eight pairs where the branch finished with fewer calls, less wall clock, or less money. Delivered is what each agent reported producing, quoted from its own closing answer, so an arm cannot look cheap by quitting early without it showing in that column.

Each prompt was run more than once per arm; the table shows the most favorable complete pairing, chosen by script. Six delivered as much as base or more. The two that did not are shown with the gap: 26 of 37 bike listings and 254 of 266 Zagreb places.

3.1  Observations and limitations

• harvest_list decided four of the eight and get_links one. The per-site toolkits decided the other three, carrying 40, 263 and 517 calls in those runs.
• Fixing one toolkit reader moved a whole pair. pxYoutube.transcript reported a closed panel as drift, so it succeeded on 12 of 226 calls; after it learned to open the panel itself, 226 of 300, and youtube#104 went from 45 unverified rows to all 162 read.
• The gains are on extraction-heavy work, not navigation-heavy work. Where a task is mostly clicking from page to page, the branch is level or slower.
• One call replaces a loop, and the loop was the cost. On facebook#108 base ran 364 execute_javascript, 332 write and 324 action calls; the branch ran three harvest_list calls that discarded 1,147 duplicate rows.
• Run-to-run variance is large. Replaying one prompt on main against main moved tool calls by up to 60%, wall clock by up to 105% and cost by up to 99%. A single pair's percentage is not readable on its own; the tool that did the work is.
• Completeness moved both ways. The branch was more complete in two pairs and less complete in two, usually by covering fewer search areas rather than by reading a page worse.
• Where the data is already structured there is nothing to win. Base reached a Reddit feed through /top.json, a YouTube channel through ytInitialData and a GitHub org through the REST API, and won or tied each time.
• Silent drift is the hard case. A stale selector returns [] and reads as an empty page, which is why the three anchors in section 2.3 exist. Across these runs the ledger reported 280 drifting calls, which main has no way to report at all.
• A stop condition can be too confident. harvest_list reported "list exhausted" after two quiet scroll steps, the agent believed it and stopped at 425 videos, where base kept reloading and reached 10,543.
• The toolkits add about 3,980 tokens to the system prompt on every run, 11.6% more than base. On short tasks that is pure overhead.
• read_form was chosen once in 27 branch runs, including on a task written for it. A tool the model does not reach for returns nothing, however well it works.