Right Question

Another sports post this week. 60 Minutes did a piece on Bill James, the Boston Red Sox data nerd. He’s credited with helping them win 2 World Series. Even if you don’t like baseball, it is an interesting data nerd piece, since Bill has developed innovative metrics to measure baseball performance. Bill says the secret is that you have to ask the right question.

As a data nerd, I appreciate his point. I have often found that the best way to answer a particular question is to answer another question. The trick is not answering either question, but tying them together to provide proper context for a decision. Every decent data nerd learns this trick, but let's consider 2 situations where I know the asked question is wrong.

First, consider direct marketing response rates. When I was in the credit card industry, I created very successful statistical response models that had over 2% in response. Today, those response rates are nearly ¼ of a percent! At 98% wrong, I was successful. Now, their model’s responses are damn near 100% wrong!

As the linked article says, these basis point response rates are still a success for the banks. And this is where the direct marketing field needs their Bill James - how can you be 100% wrong and be right?

For awhile, I’ve thought that the response distribution needs academic attention – its nearly binomial – you either responded or you didn’t – but it also has a continuous piece since each customer is a measurable profit stream. To address this special case binomial, the industry combines logistic regression (for response) with linear regression (for the profit stream). This combination results with profitable successes - that are 98% wrong.

For site selection, success is equally as complex. If a store does well, is it because of the location, the merchandise, or because of the specific store management? Is success viewed just within the one store or the network of stores in a market? ROIC may be the ultimate success measure from a real estate strategy point of view, but numerator in the equation is driven by sales that is dependant on the humans actually running the cash register, providing customer service, and selecting the merchandise.

As the economy continues to evolve, we are seeing more retailers changing their minds in their store strategies. Lots of closed stores for Talbots, Ann Taylor, Sigrid Olsen, and perhaps more are coming. How did Ann Taylor measure success? They are closing 117 out of 850 stores - that's more than 1 mistake for every 8 decisions (13.8%). That's a huge capital investment error.

What questions should we be asking? Who is our Bill James? Is it you?