Validate your Model
Validate your model
Taking into account what I understood from the explanations, these are my conclusions:
The model for chart A does not seem to be not very robust since both the light blue and the dark blue lines and the yellow light and dark lines are far appart from each other pointing to an overestimated predictive power of the model. The gains showed by the light blue line are always overestimated but a small selection of customers produce substantial gains. The probabilities of conversion do not seem in synch with the real probabilities given by the dataset. The model can be improved to be more accurate in its predictions for a given selection. The dataset seems to have the potential for more improvements in the model since the real gains line seems to allow smarter selections (20% selection for a 40% gain increase)
The model for chart B seems to deliver more robust and reliable results as compared to real gains and real probabilities (lighter and darker lines are closer to each other). A selection of 20% of customers provide a 20% increase in gains. The model should be improved since the rate at which gains are achieved through selection are relatively small from my point of view.
Am I correct with these conclusions?