How do you know it works?

I build AI systems on evidence, not vibes, and run them with my agent fleet.

My evals caught a model that wasn't ready to shipStatic data, measured 30 October 2026Link

How often each model matched my own ruling, with 95% confidence intervals.

Matched my rulingRandom pick27.6%Fine-tuned adapter (checkpoint 125)36.6%Plain Qwen52.0%Opus 5.555.5%
Show as table
My evals caught a model that wasn't ready to ship, measured 2026-10-30 Ranges in brackets are 95% confidence intervals.
SeriesMatched my ruling
Random pick134 of 486 (27.6%) (21.9%–33.3%)
Fine-tuned adapter (checkpoint 125)182 of 497 (36.6%) (31.3%–41.9%)
Plain Qwen259 of 498 (52.0%) (46.1%–57.9%)
Opus 5.5276 of 497 (55.5%) (49.7%–61.4%)
Where I overruled the agent, and where I followed itStatic data, measured 30 October 2026Link

Where I overruled the agent, the fine-tuned model matched Opus 5.5. Where I followed it, the fine-tuned model threw out good advice.

Questions where I overruled the agentFine-tuned adapter (checkpoint 125)32.6%Plain Qwen19.9%Opus 5.533.8%Questions where I followed the agent's recommendationFine-tuned adapter (checkpoint 125)43.4%Plain Qwen71.7%Opus 5.571.9%

Only 68 decisions had me overrule the agent, so those intervals are wide.

Show as table
Where I overruled the agent, and where I followed it, measured 2026-10-30 Ranges in brackets are 95% confidence intervals.
SeriesQuestions where I overruled the agentQuestions where I followed the agent's recommendation
Fine-tuned adapter (checkpoint 125)44 of 135 (32.6%) (22.2%–43.0%)132 of 304 (43.4%) (36.4%–50.3%)
Plain Qwen27 of 136 (19.9%) (12.5%–27.9%)218 of 304 (71.7%) (64.9%–78.3%)
Opus 5.546 of 136 (33.8%) (23.5%–44.1%)218 of 303 (71.9%) (65.0%–78.8%)

Headline numbers

Ruled by me
4,103
Overrule rate
31%of the time I chose something other than the agent's recommendation (3,481 of my 4,103 rulings had a recorded outcome)
Work items done
3,979
Pull requests in the shiplog
8,218
Repositories
82

Updated .

Some answers so far

  • 32.6%95% CI 22.2%–43.0%, n = 135how often my fine-tuned oracle picked what I ruled, on the decisions where I overruled the agent. Opus 5.5 did about as well, but the adapter lost to plain Qwen overall, so my evals kept it from shipping.Oracle
  • 80.0%95% CI 75.5%–84.5%, n = 200how often a judge model spotted my voice adapter's writing beside my own, with each pair judged in both orders and counted once. It spotted plain Qwen almost every time. The judge is a model, not a person.Draft Voice
  • 10.3 min → 4.2 min95% CI 9.0 min–11.6 min, n = 1,028 → 95% CI 3.5 min–5.2 min, n = 351; difference −6.1 min (95% CI −7.7 min–−4.4 min)I run my homelab like a production engineering org. This is the median time from pushing a pull request to every required check passing, before and after I sped up CI.
How often I overrule the agent's recommendationUpdated 16 November 2026Link

Of the decisions I ruled each week, the share where I chose something other than the recommended option.

0%25%50%75%this week, partialJulAugSepOctNovshort ticks on the axis: too few samples that week to show a rate

Only rulings marked as either following or departing from the recommendation are counted.

Show as table
How often I overrule the agent's recommendation, as of 2026-11-16 Ranges in brackets are 95% confidence intervals.
WeekSamplesCountedRate
Week of Jul 2710no data
Week of Aug 321no data
Week of Aug 1021911050.2% (43.7%–56.8%)
Week of Aug 17813745.7% (35.3%–56.5%)
Week of Aug 24884045.5% (35.5%–55.8%)
Week of Aug 31126no data
Week of Sep 7874450.6% (40.3%–60.8%)
Week of Sep 14712129.6% (20.2%–41.0%)
Week of Sep 211103531.8% (23.9%–41.0%)
Week of Sep 2861417829.0% (25.5%–32.7%)
Week of Oct 533911433.6% (28.8%–38.8%)
Week of Oct 1230610534.3% (29.2%–39.8%)
Week of Oct 192497329.3% (24.0%–35.3%)
Week of Oct 2645013530.0% (25.9%–34.4%)
Week of Nov 22756322.9% (18.3%–28.2%)
Week of Nov 946710322.1% (18.5%–26.0%)
Week of Nov 16 (partial)1102926.4% (19.0%–35.3%)
Architecture-violation detection accuracy against cost, by modelStatic data, measured 30 July 2026Link

How often each model found the planted problem, against what it cost per function, on 39 held-out functions.

Detection rate: real errors flagged higher is better

0%25%50%75%100%$0.0001$0.0010$0.010$0.100

False-positive rate: clean functions flagged lower is better

0%25%50%75%100%$0.0001$0.0010$0.010$0.100
  • DeepSeek V4 Flash, ten calls
  • DeepSeek V4 Flash, one call
  • Claude Sonnet 5, ten calls
  • Claude Sonnet 5, one call
  • GPT-5.6 Terra, medium effort, ten calls
  • GPT-5.6 Terra, medium effort, one call

The cost axis is logarithmic: each gridline is ten times the last.

Answers are graded against a fixed keyword answer key, not by another model.

The differences between models are not statistically significant.

Show as table
Architecture-violation detection accuracy against cost, by model, measured 2026-07-30 Ranges in brackets are 95% confidence intervals.
SetupCost (US dollars)Detection rate: real errors flagged (higher is better)False-positive rate: clean functions flagged (lower is better)
DeepSeek V4 Flash, ten calls$0.001920 of 23 (87.0%) (67.9%–95.5%)3 of 16 (18.8%) (6.6%–43.0%)
DeepSeek V4 Flash, one call$0.000813 of 23 (56.5%) (36.8%–74.4%)1 of 16 (6.3%) (1.1%–28.3%)
Claude Sonnet 5, ten calls$0.03815 of 23 (65.2%) (44.9%–81.2%)2 of 16 (12.5%) (3.5%–36.0%)
Claude Sonnet 5, one call$0.02316 of 23 (69.6%) (49.1%–84.4%)2 of 16 (12.5%) (3.5%–36.0%)
GPT-5.6 Terra, medium effort, ten calls$0.02620 of 23 (87.0%) (67.9%–95.5%)1 of 16 (6.3%) (1.1%–28.3%)
GPT-5.6 Terra, medium effort, one call$0.004716 of 23 (69.6%) (49.1%–84.4%)1 of 16 (6.3%) (1.1%–28.3%)
Agent activityUpdated 16 November 2026Link

Subagents launched each finished week, and the most running at once that week.

01K2K3K4KSubagents launchedMost running at once015304560JulAugSepOctNov
  • Subagents launched
  • Most running at once (right axis)
Show as table
Agent activity, as of 2026-11-16
WeekSubagents launchedMost running at once
Week of Jul 1330523
Week of Jul 201,77123
Week of Jul 273,14423
Week of Aug 32,94642
Week of Aug 102,84036
Week of Aug 1781226
Week of Aug 241,29222
Week of Aug 3156423
Week of Sep 768425
Week of Sep 1451229
Week of Sep 2190228
Week of Sep 281,61241
Week of Oct 51,14128
Week of Oct 121,33640
Week of Oct 191,36940
Week of Oct 261,77937
Week of Nov 21,51529

What connects to what

Show
  • uses
  • related project
  • project
  • technology
Arrange
Show every summary as a list

Technologies

  • Claude

    While I am working on replacing Claude Code with my own development system and harness specifically trained for my coding style for now I often use Claude Code.

    Claude is my main driver right now I do also use Codex for longer running tasks, particularly computer use tasks.

    The main portion of Claude's use is for my "fleet" see that page for more details about the fleet and how its decision system enables me to build faster.

  • Effect

    Effect.ts is one of the main components of all my software. Effect ports a lot of ZIO from Scala to Typescript bringing many functional paridigms.

    In my opinion functional programming is one of the key enablers for effective LLM usage. For example, side Effects, a function making changes in an unrelated section of code, degrade LLM's performance even more than humans. Humans can remember many things from a long time in the past, while LLM's can only know whats in their context window. So the more disparate areas of the code that the model has to know means that it fills up its context window faster. This both reduces the size of the code that the LLM can work on it also degrades the LLM's prerformance. (see context rot) So the less you can fill the LLM's context window the better.

    This is where making small isolated composable functions reduces the amount the model needs in its context window.

    I also find that Effect.ts enables other benefits in very convenient ways, as the Effect.ts docs explain:

    When coding agents write a growing share of your software, the bottleneck moves from writing code to trusting it. Effect makes a program’s failure modes, dependencies, and lifecycle visible to the compiler, turning runtime surprises into precise feedback an agent can act on. And when the thing you are building is itself an AI application, flaky providers, retries, streaming, and rate limits are exactly the problems Effect solves out of the box. Effect Intro

    Further reading:

  • Fine-tuning

    Fine-tuning is used in various projects to enable me to get frontier level performance for very specific tasks on cheap models I can run in my homelab.

  • PostgreSQL

    While I reach for graph databases as my default there is a familiarity and simplicity that comes with a SQL database. I have settled on PostgreSQL as my SQL database of choice because of its resource efficiency and my own familiarity with it from previous jobs.

  • Python

    Python has been the single language that I have used the most throughout my career. While my recent projects are mostly Typescript I do still use python for my machine learning projects. I do find that my familiarity with Python matters less now that LLM's do the coding but for those rare bugs it is nice to go and read code that I am familiar enough with to easily scan.

    I used Python as my main language from 2015-2019 at Hewlett Packard Enterprise and Sumato Homes. Then from 2022-2025 at Chegg.

  • Quint

    While I am less familiar with formal verification I am slowly learning as I have my agents create formal static analysis code in quint to verify that I won't reach deadlocks.

    My overall philosophy of making deterministic checks applies here as the closer I can make my quint match my code the more that I can feel safe to merge agent code without human review.

  • Self-hosted LLM

    I use Qwen 3.8 27b as my main self hosted model. Right now I only have 2x AMD ai pro r9700's which brings me to a total of 64gb of vram. To keep performance for coding I don't go lower than FP8 quantization which only leaves ~ 30gb for kv cache and Lora adapters. This is served via some customized Ubuntu VM images I build and bake the AMD GPU stack in. I use vLLM as my run time although because of the lack of support for the r9700's in tensor parallelism I do have LLMs maintaining my own version of vLLM. I have not reviewed their changes and so have not pushed those changes upstream.

  • TypeScript

    While my favorite language to code in by hand used to be Kotlin, I have shifted to typescript.

    I adore functional programming and find the advances in languages like Haskell facinating. However the interoperability and lack of widespread corporate deployment has prevented me from learning Haskell as much as I would like. That is where Effect.ts made typescript so enticing, I could use the wonderful ZIO system while also using a language that shows up all over the place.

    Further reading:

Projects

Projects

  • tooling2026

    Agent Fleet

    The organized fleet that enables me to code faster as I am only needed for key decisions

  • tooling2026

    API Proxy

    The secrets proxy so no agent ever holds a secret or token

  • agents2026

    Decision queue

    Every judgment call is a record.

    31%

  • tooling2026

    Architecture checks

    Static checks for the architecture, not just the syntax.

  • tooling2026

    Workstation Config

    Consistent configuration for all workstations

  • evaluation2026

    Draft Voice

    An adapter that drafts in my own voice.

  • research2026

    Research

    Self-reports and what they reflect.

  • tooling2026

    Application tracker

    A job-application tracker with a Firefox extension that fills in forms.

Photographs

The photographs open on the About page.

Photographs

Writing

Longer notes on how I check things live on the blog.

Every post, by section