Uplift

Adaptive models can make actions more relevant to the customer by calculating personalized propensities, a principle that potentially increases the success rate of the action. This uplift is the increase in success rate of model-based actions when compared to random actions and is a valuable business metric. Next-best-action Designer automatically creates a control group of customers that is offered random actions to allow assessment of the uplift of adaptive models.

If the system does not use next-best-action Designer, the same can be accomplished by building a decision strategy. This lesson illustrates a decision strategy design pattern that allows you to measure the uplift of your models.

In its basic form, a Next-Best-Action (NBA) strategy would look like this:

Individual actions are imported via the blue Proposition Data decision component and enhanced with a propensity calculated by the Prediction Strategy component. The green Prioritize component selects the action with the highest propensity.

Strategies are usually further enhanced with concepts such as eligibility criteria and business rules, but the focus of this lesson is on the adaptive models used in the underlying Prediction Strategy:

The actions from the NBA strategy are used as external input for the Prediction Strategy. They are propagated through the Adaptive Model component and the resulting propensity for each action is propagated back to the NBA strategy.

If you want to measure the uplift of the adaptive models you will need to use a control group with a random action that contains a small percentage of your customers. This will allow you to compare the success rate of the actions prioritized by the models with an unbiased benchmark. In the strategy you could do this by adding a component in the Prediction Strategy that sets the propensity randomly for 10% of the decisions.

Consider the following Prediction Strategy:

The strategy imports the actions from the NBA strategy and propagates them through the Adaptive Model component. From the adaptive model component, the propositions are routed through two branches:

• In the first (the ‘model route’), the propensity determined by the adaptive model is propagated.
• In the second (the ‘random route’), a random propensity value is propagated instead.

The red Champion Challenger component is used to select the model route in 90% of the decisions and the random route for 10% of the decisions. The propensity for each action is propagated back to the NBA strategy.

Note that with the design pattern of this Prediction Strategy, the adaptive models can learn from all responses, including responses to random actions. This may add value as it feeds the models with all possible actions and responses, supporting model adaptivity.

To offset the impact of the model-based actions to those based on random actions, you model a report based upon the strategy property that reflects the use of either the model route or the random route when determining the next best action.

In this case, the uplift (the increase in success rate compared to random action) generated by the adaptive models is around 40%.

When business objectives require the random actions to be consistently taken with a consistent segment of the customer base, you can use customer properties (i.e. the customer ID) to determine the routing through the Prediction strategy.

In this design pattern, the red Switch component selects either the model route or the random route. The selection is based on a hash code derived from the last two digits of the customer ID, which is used by the Logic component to segment the customers.

This approach works well if the digits have no implicit meaning. If they do (i.e. when the last digits of the ID are 99 if the customer was from an acquired company), this type of customer would either never or always get a random action. It is best practice to check that the digits have a (roughly) uniform distribution. 

Uplift is a valuable business metric. To measure the uplift of your models, you challenge the models with random actions, and track the source of the propensity so that you can report on it later. Hence including random actions in a Prediction Strategy makes the adaptive power of the adaptive models measurable.