Keep a Ledger for Every Experiment

An experiment note is not a diary; it is a small contract covering the hypothesis, cost, evidence, and next decision.

阅读中文版 →
On this page

As experiments accumulate, context disappears before code does. It soon becomes difficult to remember why a feature changed, why one parameter was abandoned, or whether a failed result came from the direction or the implementation.

I now treat each experiment as an entry that must be settled: what was invested, what evidence was produced, and what belief changed?

A minimum experiment record

The note can be short, but the hypothesis should exist before the run begins.

## Problem
What is happening, and why does it matter?

## Hypothesis
If X changes, Y should improve because Z.

## Variables
What changed, and what remained fixed?

## Results
Primary metrics, guardrails, cost, and anomalies.

## Decision
Supported, rejected, or still uncertain?

## Next step
Continue, stop, narrow the scope, or collect new evidence?

The purpose is to turn “we tried something” into “we tested a claim that could be disproved.”

Preserve the boundaries

Results depend on data versions, time windows, code commits, random seeds, model versions, runtime configuration, and evaluation scripts. These details look mechanical, but they determine whether a result can be reproduced. A conclusion without its boundary conditions is usually only an impression.

Settle failed experiments too

Failed work is easy to omit because there is no gain to present. Without a record, the same path may be repeated months later. A useful failure note explains why the hypothesis was unsupported, what uncertainty remains, when the idea might still work, and why the work stopped.

Failure becomes knowledge only after it is interpreted and preserved.

Let the ledger influence decisions

The ledger should eventually reveal patterns: features that improve only offline, recurring data problems, or expensive experiments that produce little information. Once those patterns change the design of the next experiment, the record begins to compound.

RELATED WRITING

DISCUSSION

Discussion