Evaluating RAG Beyond Plausible Answers
Decompose RAG failures across retrieval, context construction, and generation so each result points to an actionable change.
Read noteWRITING
Technical notes, paper readings, and the occasional aside.
Decompose RAG failures across retrieval, context construction, and generation so each result points to an actionable change.
Read noteOnce an agent calls tools, changes state, and replans, conventional request logs are no longer enough to explain success or failure.
Read noteOffline metrics are evidence, not a conclusion; data splits, exposure bias, and user segments determine whether that evidence is trustworthy.
Read noteI want a record of learning, engineering judgement, and long-term thinking that can be revisited and tested.
Read noteA close reading of scaled dot-product attention, multi-head attention, positional encoding, and the engineering conditions that made Transformer consequential.
Read noteA useful learning note is not a tutorial paraphrase. It keeps the question, the experiment path, and the boundary of the conclusion together.
Read noteAn experiment note is not a diary; it is a small contract covering the hypothesis, cost, evidence, and next decision.
Read noteHow shared embeddings connect the FM and deep branches, and why the architecture remains a useful recommender-systems baseline.
Read noteDepth should follow frequency, the cost of being wrong, and whether the technology forms part of your core judgment.
Read noteMore useful than remembering that a metric improved is keeping how the problem was defined, how the experiment was designed, and where the conclusion stops.
Read noteHow the exposure-click-conversion probability chain helps ESMM address sample-selection bias and sparse supervision in CVR estimation.
Read noteHow interleaved Thought, Action, and Observation trajectories connect language reasoning to tools—and what production agents still need beyond the prompt.
Read noteAn engineering experiment is not a product plan. It records hypotheses, prototypes, and explorations that do not yet have an answer.
Read note