Hypothesis, sample size, statistical power, conflicting metrics and AI features. One experiment, worked end to end, with every number shown.
"Don't ship v1. Ship the learning" is the line I would keep. I write the decision rule down before an agent pilot starts too, so nobody rewrites it once the results come in.
Yes, that should be done no matter what. Tweaking the hypothesis according to the results can be dangerous at times.
"Don't ship v1. Ship the learning" is the line I would keep. I write the decision rule down before an agent pilot starts too, so nobody rewrites it once the results come in.
Yes, that should be done no matter what. Tweaking the hypothesis according to the results can be dangerous at times.