How do you know it works?
I build AI systems on evidence, not vibes, and run them with my agent fleet.
How often each model matched my own ruling, with 95% confidence intervals.
Show as table
| Series | Matched my ruling |
|---|---|
| Random pick | 134 of 486 (27.6%) (21.9%– |
| Fine-tuned adapter (checkpoint 125) | 182 of 497 (36.6%) (31.3%– |
| Plain Qwen | 259 of 498 (52.0%) (46.1%– |
| Opus 5.5 | 276 of 497 (55.5%) (49.7%– |
Where I overruled the agent, the fine-tuned model matched Opus 5.5. Where I followed it, the fine-tuned model threw out good advice.
Only 68 decisions had me overrule the agent, so those intervals are wide.
Show as table
| Series | Questions where I overruled the agent | Questions where I followed the agent's recommendation |
|---|---|---|
| Fine-tuned adapter (checkpoint 125) | 44 of 135 (32.6%) (22.2%– | 132 of 304 (43.4%) (36.4%– |
| Plain Qwen | 27 of 136 (19.9%) (12.5%– | 218 of 304 (71.7%) (64.9%– |
| Opus 5.5 | 46 of 136 (33.8%) (23.5%– | 218 of 303 (71.9%) (65.0%– |
Headline numbers
- Ruled by me
- 4,103
- Overrule rate
- 31%of the time I chose something other than the agent's recommendation (3,481 of my 4,103 rulings had a recorded outcome)
- Work items done
- 3,979
- Pull requests in the shiplog
- 8,218
- Repositories
- 82
Updated .
Some answers so far
- 32.6%95% CI 22.2%–43.0%, n = 135how often my fine-tuned oracle picked what I ruled, on the decisions where I overruled the agent. Opus 5.5 did about as well, but the adapter lost to plain Qwen overall, so my evals kept it from shipping.Oracle
- 80.0%95% CI 75.5%–84.5%, n = 200how often a judge model spotted my voice adapter's writing beside my own, with each pair judged in both orders and counted once. It spotted plain Qwen almost every time. The judge is a model, not a person.Draft Voice
- 10.3 min → 4.2 min95% CI 9.0 min–11.6 min, n = 1,028 → 95% CI 3.5 min–5.2 min, n = 351; difference −6.1 min (95% CI −7.7 min–−4.4 min)I run my homelab like a production engineering org. This is the median time from pushing a pull request to every required check passing, before and after I sped up CI.
Of the decisions I ruled each week, the share where I chose something other than the recommended option.
Only rulings marked as either following or departing from the recommendation are counted.
Show as table
| Week | Samples | Counted | Rate |
|---|---|---|---|
| Week of Jul 27 | 1 | 0 | no data |
| Week of Aug 3 | 2 | 1 | no data |
| Week of Aug 10 | 219 | 110 | 50.2% (43.7%– |
| Week of Aug 17 | 81 | 37 | 45.7% (35.3%– |
| Week of Aug 24 | 88 | 40 | 45.5% (35.5%– |
| Week of Aug 31 | 12 | 6 | no data |
| Week of Sep 7 | 87 | 44 | 50.6% (40.3%– |
| Week of Sep 14 | 71 | 21 | 29.6% (20.2%– |
| Week of Sep 21 | 110 | 35 | 31.8% (23.9%– |
| Week of Sep 28 | 614 | 178 | 29.0% (25.5%– |
| Week of Oct 5 | 339 | 114 | 33.6% (28.8%– |
| Week of Oct 12 | 306 | 105 | 34.3% (29.2%– |
| Week of Oct 19 | 249 | 73 | 29.3% (24.0%– |
| Week of Oct 26 | 450 | 135 | 30.0% (25.9%– |
| Week of Nov 2 | 275 | 63 | 22.9% (18.3%– |
| Week of Nov 9 | 467 | 103 | 22.1% (18.5%– |
| Week of Nov 16 (partial) | 110 | 29 | 26.4% (19.0%– |
How often each model found the planted problem, against what it cost per function, on 39 held-out functions.
Detection rate: real errors flagged higher is better
False-positive rate: clean functions flagged lower is better
- DeepSeek V4 Flash, ten calls
- DeepSeek V4 Flash, one call
- Claude Sonnet 5, ten calls
- Claude Sonnet 5, one call
- GPT-5.6 Terra, medium effort, ten calls
- GPT-5.6 Terra, medium effort, one call
The cost axis is logarithmic: each gridline is ten times the last.
Answers are graded against a fixed keyword answer key, not by another model.
The differences between models are not statistically significant.
Show as table
| Setup | Cost (US dollars) | Detection rate: real errors flagged (higher is better) | False-positive rate: clean functions flagged (lower is better) |
|---|---|---|---|
| DeepSeek V4 Flash, ten calls | $0.0019 | 20 of 23 (87.0%) (67.9%– | 3 of 16 (18.8%) (6.6%– |
| DeepSeek V4 Flash, one call | $0.0008 | 13 of 23 (56.5%) (36.8%– | 1 of 16 (6.3%) (1.1%– |
| Claude Sonnet 5, ten calls | $0.038 | 15 of 23 (65.2%) (44.9%– | 2 of 16 (12.5%) (3.5%– |
| Claude Sonnet 5, one call | $0.023 | 16 of 23 (69.6%) (49.1%– | 2 of 16 (12.5%) (3.5%– |
| GPT-5.6 Terra, medium effort, ten calls | $0.026 | 20 of 23 (87.0%) (67.9%– | 1 of 16 (6.3%) (1.1%– |
| GPT-5.6 Terra, medium effort, one call | $0.0047 | 16 of 23 (69.6%) (49.1%– | 1 of 16 (6.3%) (1.1%– |
Subagents launched each finished week, and the most running at once that week.
- Subagents launched
- Most running at once (right axis)
Show as table
| Week | Subagents launched | Most running at once |
|---|---|---|
| Week of Jul 13 | 305 | 23 |
| Week of Jul 20 | 1,771 | 23 |
| Week of Jul 27 | 3,144 | 23 |
| Week of Aug 3 | 2,946 | 42 |
| Week of Aug 10 | 2,840 | 36 |
| Week of Aug 17 | 812 | 26 |
| Week of Aug 24 | 1,292 | 22 |
| Week of Aug 31 | 564 | 23 |
| Week of Sep 7 | 684 | 25 |
| Week of Sep 14 | 512 | 29 |
| Week of Sep 21 | 902 | 28 |
| Week of Sep 28 | 1,612 | 41 |
| Week of Oct 5 | 1,141 | 28 |
| Week of Oct 12 | 1,336 | 40 |
| Week of Oct 19 | 1,369 | 40 |
| Week of Oct 26 | 1,779 | 37 |
| Week of Nov 2 | 1,515 | 29 |
What connects to what
- uses
- related project
- project
- technology
Show every summary as a list
Technologies
Claude
While I am working on replacing Claude Code with my own development system and harness specifically trained for my coding style for now I often use Claude Code.
Claude is my main driver right now I do also use Codex for longer running tasks, particularly computer use tasks.
The main portion of Claude's use is for my "fleet" see that page for more details about the fleet and how its decision system enables me to build faster.
Effect
Effect.ts is one of the main components of all my software. Effect ports a lot of ZIO from Scala to Typescript bringing many functional paridigms.
In my opinion functional programming is one of the key enablers for effective LLM usage. For example, side Effects, a function making changes in an unrelated section of code, degrade LLM's performance even more than humans. Humans can remember many things from a long time in the past, while LLM's can only know whats in their context window. So the more disparate areas of the code that the model has to know means that it fills up its context window faster. This both reduces the size of the code that the LLM can work on it also degrades the LLM's prerformance. (see context rot) So the less you can fill the LLM's context window the better.
This is where making small isolated composable functions reduces the amount the model needs in its context window.
I also find that Effect.ts enables other benefits in very convenient ways, as the Effect.ts docs explain:
When coding agents write a growing share of your software, the bottleneck moves from writing code to trusting it. Effect makes a program’s failure modes, dependencies, and lifecycle visible to the compiler, turning runtime surprises into precise feedback an agent can act on. And when the thing you are building is itself an AI application, flaky providers, retries, streaming, and rate limits are exactly the problems Effect solves out of the box. Effect Intro
Further reading:
Fine-tuning
Fine-tuning is used in various projects to enable me to get frontier level performance for very specific tasks on cheap models I can run in my homelab.
PostgreSQL
While I reach for graph databases as my default there is a familiarity and simplicity that comes with a SQL database. I have settled on PostgreSQL as my SQL database of choice because of its resource efficiency and my own familiarity with it from previous jobs.
Python
Python has been the single language that I have used the most throughout my career. While my recent projects are mostly Typescript I do still use python for my machine learning projects. I do find that my familiarity with Python matters less now that LLM's do the coding but for those rare bugs it is nice to go and read code that I am familiar enough with to easily scan.
I used Python as my main language from 2015-2019 at Hewlett Packard Enterprise and Sumato Homes. Then from 2022-2025 at Chegg.
Quint
While I am less familiar with formal verification I am slowly learning as I have my agents create formal static analysis code in quint to verify that I won't reach deadlocks.
My overall philosophy of making deterministic checks applies here as the closer I can make my quint match my code the more that I can feel safe to merge agent code without human review.
Self-hosted LLM
I use Qwen 3.8 27b as my main self hosted model. Right now I only have 2x AMD ai pro r9700's which brings me to a total of 64gb of vram. To keep performance for coding I don't go lower than FP8 quantization which only leaves ~ 30gb for kv cache and Lora adapters. This is served via some customized Ubuntu VM images I build and bake the AMD GPU stack in. I use vLLM as my run time although because of the lack of support for the r9700's in tensor parallelism I do have LLMs maintaining my own version of vLLM. I have not reviewed their changes and so have not pushed those changes upstream.
TypeScript
While my favorite language to code in by hand used to be Kotlin, I have shifted to typescript.
I adore functional programming and find the advances in languages like Haskell facinating. However the interoperability and lack of widespread corporate deployment has prevented me from learning Haskell as much as I would like. That is where Effect.ts made typescript so enticing, I could use the wonderful ZIO system while also using a language that shows up all over the place.
Further reading:
Projects
Agent Fleet
The organized fleet that enables me to code faster as I am only needed for key decisions
API Proxy
The secrets proxy so no agent ever holds a secret or token
Decision queue
Every judgment call is a record.
Architecture checks
Static checks for the architecture, not just the syntax.
Workstation Config
Consistent configuration for all workstations
Draft Voice
An adapter that drafts in my own voice.
Research
Self-reports and what they reflect.
Application tracker
A job-application tracker with a Firefox extension that fills in forms.
Projects
Agent Fleet
The organized fleet that enables me to code faster as I am only needed for key decisions
API Proxy
The secrets proxy so no agent ever holds a secret or token
Decision queue
Every judgment call is a record.
31%
Architecture checks
Static checks for the architecture, not just the syntax.
Workstation Config
Consistent configuration for all workstations
Draft Voice
An adapter that drafts in my own voice.
Research
Self-reports and what they reflect.
Application tracker
A job-application tracker with a Firefox extension that fills in forms.
Writing
Longer notes on how I check things live on the blog.
- Writings
How I design: correct the first draft, then decide
I design with AI agents. Their first draft is where my thinking starts, not where it ends. Four small decisions from one evening show how.
- Writings
I don't watch my agents work. I rule and I review.
How I run a fleet of AI agents without watching them: they bank judgment calls, I rule in batches, and enforced checks hold the rules.