"Deming's Funnel, and Why Your Team Learns From Randomness"
W. Edwards Deming used to run a demonstration at his seminars with a funnel, a marble, and a table with a target drawn on it. The setup was simple. Drop the marble through the funnel. It lands slightly off the target. Now adjust the funnel to compensate for the miss. Drop again. Adjust again.
The intuition says the corrections should bring the marble closer to the target. The mathematics says the opposite. Adjusting the funnel after every drop roughly doubles the variance compared to leaving the funnel fixed over the target. Every correction adds a new source of error, because you are adjusting for the last deviation, which was itself partly random, and the adjustment is then subject to the same random forces that produced the deviation you were trying to fix.
Deming called it tampering. He considered it one of the most common management errors in American industry, and he spent the last two decades of his career trying to get people to stop doing it.
Why the demonstration works
The funnel experiment relies on a distinction that sounds academic until you've been in a room where someone ignores it: the distinction between a stable process and an unstable one.
A stable process has variation, but the variation is consistent and predictable. The marble doesn't hit the target every time, but the size and shape of the misses follow a pattern. The misses have a normal range. Walter Shewhart, Deming's mentor at Bell Labs in the 1920s, called this common cause variation, the inherent noise of the system doing its ordinary work.
An unstable process has something else going on. A new input, a changed condition, a broken component. Shewhart called this special cause variation, because it has a specific assignable cause that can be found and removed.
The funnel rule is: if the process is stable, leave it alone. If something has genuinely shifted, find the special cause and address it. Telling those two situations apart is the entire skill, and almost nobody checks which one they're in before reacting.
What tampering looks like in analytics
Every company I've worked in tampers with its metrics.
Conversion rate drops 4% week over week, so somebody changes the landing page. The following week conversion is up 3% and the change gets credit. The week after that it drops 5% and a second change goes in. Nobody ever established what normal week-to-week movement looks like for that metric, so there is no way to tell whether 4% was a signal or the ordinary bounce of a rate computed from a small denominator over a short window.
The cost is not only the wasted effort on the page change, though that cost is real. It's that the organization is now learning from randomness. Practices get adopted because they happened to precede an up-week. Theories get built on movement that had no cause. Those theories become the basis for the next decision, and the one after that, and the organization develops a confident body of knowledge that is, to a meaningful degree, fiction.
I've watched a marketing team at one company run a full quarter of A/B tests where every result was within the normal week-to-week noise band. They shipped four "winning" variants, wrote a best-practices document, and trained the next cohort of hires on the findings. None of the results were statistically significant. Nobody checked.
The control chart solution
Deming's fix was not complicated. It was a control chart, a tool Shewhart invented in 1924 at Bell Labs. Plot the metric over time. Calculate the average and the upper and lower control limits, typically set at three standard deviations from the mean. Any point inside the limits is common cause variation. Leave it alone. Any point outside the limits, or a pattern like seven consecutive points on one side of the average, is a signal that something changed.
The control chart is not a sophisticated statistical tool. It's arithmetic. Mean, standard deviation, multiply by three, draw two lines. The value is not in the math. It's in having the lines drawn before the metric moves, so when it does move, the conversation starts with "is this inside the normal range" rather than "what happened."
For most business metrics, building a control chart takes less than an hour. Pull the weekly values for the past year. Compute the average and the standard deviation. Set the limits. Plot it. Most teams discover that half of what they've been reacting to sits comfortably inside the normal range, and that the genuine shifts, the ones outside the limits, were buried in the noise of all the reactions to normal variation.
The cost of not checking
The direct cost is wasted work. A landing page change, a pricing adjustment, a campaign launch, each initiated in response to movement that was going to happen regardless. Those changes consume design time, engineering time, and management attention.
The indirect cost is worse. An organization that reacts to noise develops a superstitious relationship with its metrics. Patterns emerge where none exist. Rituals develop around metric reviews. The Friday meeting becomes a weekly exercise in explaining random variation with business narratives, and the narratives get more elaborate as the team practices them.
Meanwhile, the real signals compete for attention with the noise reactions. A true shift in customer behavior gets the same response as last week's random bounce: a theory, a meeting, a change, a review next week. The signal gets the same treatment as the noise, which means it gets no special treatment at all.
Why nobody does this
The reason is not mathematical. It's political. Sitting with a bad-looking number for a week and saying "that's within the normal range, we're not changing anything" is uncomfortable. Telling a VP that the 4% drop they flagged in the Monday email is not a problem, it's Tuesday, and the metric does this every few weeks, is a conversation most analysts avoid.
Deming spent forty years making this argument. The funnel experiment was his way of showing it in a room, in ten minutes, with a marble. The marble always convinced people. Applying it to their own metrics, where the stakes felt higher and the pressure to act was real, was the part that took the rest of his career.
The metric that moved last week probably moved because metrics move. Check the range before you check the cause.