Calibration Woes: Part II

In a previous post I talked about an estimator which was calibrated retrospectively but appeared extremely miscalibrated prospectively. The historical data for calibration was produced using a different decision procedure, whereas the recent data was the result of decisions produced by using the model in conjunction with a linear program. I had some questions about why this was happening, whether I could correct the calibration, and whether I should correct the calibration.

Well definitely I can correct the calibration. I fit the (mis)calibration plot to a spline and used that to define a new estimator which was the old estimator followed by a nonlinear output correction. I ran both the original and recalibrated models in production for a while and checked the calibration of each: here's the results.

Red = original, Green = recalibrated

So my fears about the linear program fighting against my calibration correction appear unfounded: it looks like my correction went through essentially unmodified.

With respect to why the model was miscalibrated, perhaps my suspicion of reductions to regression has made me overly quick to blame the problem on the sinister LP. Maybe the calibration was just screwed up initially, or maybe there is a bug in the model implementation somewhere.

On the most important question of whether the calibration should be corrected, I'm sad to say that there is almost no change in any important business metric with respect to usage of the original versus recalibrated model.  We're going to switch over to the recalibrated model anyway, because it feels better.