Monitoring predictions

4 タスク

15 分

Pega Customer Decision Hub 8.6

Pega Customer Decision Hub 8.8 Pega Customer Decision Hub 8.7 Pega Customer Decision Hub 8.5 Pega Customer Decision Hub 8.4

Visible to: All users

初級 Pega Customer Decision Hub 8.6 英語

シナリオ

U+ Bank has recently implemented Pega Customer Decision Hub™ to display a personalized credit card offer to eligible customers on their website. The business requests a report on the boost in success rate that the artificial intelligence (AI) generates. As a data scientist, you inspect the Predict Web Propensity prediction that aims to optimize the click through rate of the web banners that contain the credit card offers.

Use the following credentials to log in to the exercise system:

Role	User name	Password
Data scientist	DataScientist	rules

Your assignment consists of the following tasks:

Task 1: Inspect the control group settings

Inspect the settings for the control group by answering the following questions:

1. Is the control group based on a customer attribute?
2. Will the control group be large enough to measure the lift?

Task 2: Inspect the success rate graph

Inspect the success rate of the Control group and the Test group over time by answering the followings questions:

1. Is the success rate of the Control group stable over time? Is this expected?
2. Is the success rate of the Test group higher or lower than the success rate of the Control group? Is this expected?

Task 3: Inspect the lift graph

Inspect the lift over time by answering the following questions:

1. What is the current boost in the success rate that the AI generates?
2. What is the trend of the lift?

Task 4: Inspect the performance graph

Inspect the performance of the AI by answering the following questions:

1. What is the current performance of the prediction?
2. What is the current distribution of the prediction?
3. Which of the credit card offers have a low success rate?

このチャレンジを完了するには、Pegaインスタンスを起動する必要があります。

起動には5分ほどかかることがありますので、しばらくお待ちください。

• ログイン

チャレンジ ウォークスルー

詳細なタスク

1 Inspect the control group settings

1. Log in as Data Scientist with user name DataScientist and password rules.
2. In the navigation pane on the left, click Intelligence > Prediction Studio.
3. Click the Predict Web Propensity tile to open the prediction configuration.
   

Question - Is the control group based on a customer attribute?

4. Click the Settings tab.
   

ヒント： By default, the Control group is based on a percentage. Based on best practices, the Control group is set to 2% of the customers. Alternatively, you can base the Control group on a customer attribute by selecting Field.

Question - Will the control group be large enough to measure lift?

ヒント： In a web scenario, with high volumes of customer interaction, 2% is sufficient to measure lift.

2 Inspect the success rate graph

1. Click the Analysis tab.
2. Inspect the success rate graph.

Question - Is the success rate of the Control group stable over time? Is this expected?

ヒント： The success rate of the Control group is dependent on customer behavior and engagement policies, so it may change over time.

Question - Is the success rate of the Test group higher or lower than the success rate of the Control group? Is this expected?

ヒント： The success rate of the Test group is expected to be higher than the success rate of the Control group as AI calculates what credit card offer has the highest likelihood of a click-through. The credit card offer is random in the Control group.

3 Inspect the lift graph

Question - What is the current boost in success rate that the AI generates?

1. Inspect the lift graph.
   

ヒント： Hover over a date to display the lift. The current lift is also shown in the upper left of the graph: 103.72%.

Question - What is the trend of the lift?

ヒント： As U+ Bank implemented Pega Decision Management recently, there is not enough data to detect a trend yet. Once sufficient data is collected, the system generates notifications when there is no lift or when the lift drops significantly.

4 Inspect the performance graph

Question - What is the current performance of the prediction?

1. Inspect the performance graph.
   

ヒント： Hover over a date to display the performance. The current performance is also shown in the upper left of the graph: 84.63 (AUC).

Question - What is the current distribution of the prediction?

1. Click Show distribution.
   

ヒント： The distribution can be displayed as a ROC curve and a decile distribution for further inspection.

Question - Which of the credit card offers have a low success rate?

1. In the navigation pane on the left, click Models.
2. Click the Web_Click_Through_Rate tile to open the model configuration.
3. On the Monitor tab, inspect the performance graph.
   

ヒント： The green StandardCard has a lower success rate than the other cards. It also has a low performance, and inspection of the predictors that the adaptive models use, which drive the Prediction, might improve the prediction lift.

---

---

このモジュールは、下記のミッションにも含まれています。

• Data Scientist v3