I’m an engineer, so I understand that the green line is not a good number by which to measure a solution. In other words, it’s a poor engineering metric. However, I’m also a marketer, so I understand that users don’t care a whit about engineering. That’s why the green line is the best one to put on the website graph, because it’s the one that shows users the information they expect to see on that graph. You said as much yourself, “that users don’t think of things in this way… They look at what is in their inbox.”

Now, Google is great at what it does, and you guys fighting spam are doing us all a wonderful service. But that’s not the take-away here. The take-away here is to be careful what you communicate. When you put two lines on the same line graph, people expect them to be measuring the same thing. And when they aren’t, people can feel cheated–especially if the graph makes you appear better than you would otherwise–and it doesn’t matter how good or poor an engineering metric it is, because people don’t care about engineering metrics. They care about perception.

-TimK

The data on the charts we presented is the same data we use internally to measure outselves and fight spam. Spam filters are measured by their “false negative” rate: that is how much spam escapes them. A simple way to think of the formula is this:

total spam = spam filtered automatically + spam reported by users
miss rate = spam reported by users / total spam

If we make our filter better, then our miss rate should decline.

We understand that users don’t think of things in this way though. They look at what is in their inbox. But using that as a metric is not useful. Consider this formula:

bad spam metric = spam reported by users / total mail in inbox

This number can go down simply because more non-spam (good) mail arrives in inboxes. It tells you nothing about spam filter performance.

You can think of it like a coffee filter. The miss rate is how many coffee grains escape through the filter versus how much coffee was put in the filter originally. You wouldn’t measure the coffee filter by how many grains are in the coffee pot vs. how much water is in the coffee pot. Because then I could just pour a bunch of water in the pot and say since there are very few grains of coffee relative to the water in the pot, the filter must be very good.

Hopefully, this clears things up for you.

Brad Taylor
Gmail anti-spam engineering technical lead