How Bad is Bad? 3 Methods to Analyze Based on Different Business Objectives

Same metric, different business objectives, different methods to analyze

When analyzing a business metric, have you ever faced an open-ended question like: How bad is too bad? Is it that bad that we need to improve? How do we define bad?

For example, in third-party logistics (3PL) business, we care about delivery time. The faster the better. Therefore, how do we determine how long is a ‘late’ delivery time?

1. Using percentile distribution

Objective: We are offering our delivery service for another business, and we promise a 95% service-level agreement (SLA), that 95% of orders will be delivered on time, otherwise we will be fined.

We need to analyze how long the delivery time of the 5% longest orders is, and we should not promise the SLA to be earlier than that. Looking at the historical data below, the 5% longest orders had delivery time >58 mins. Assuming the future condition will be similar to the past occurrences, we should be able to promise that the remaining 95% of orders are able to be delivered below 58 mins. Therefore, we can have the promised SLA for ≤58 mins.

Table 1. Distribution of order delivery time from a 3PL delivery service

2. Using outlier formula

Objective: We want to educate drivers who deliver orders way later than other drivers, therefore we want to know who are the drivers with delivery time exceeding outliers.

As we want to identify drivers with late delivery time that exceeds outliers, then we need to calculate the Upper Outlier. From the dataset in Table 1, Upper Outlier is calculated as below:

• Q1 (25th percentile) = 37.5 mins
• Q3 (75th percentile) = 52.3 mins
• IQR = 52.3 mins – 37.5 mins = 14.8 mins
• Upper Outlier = 52.3 mins + (1.5 x 14.8 mins) = 74.5 mins

This means that drivers that deliver order in more than 74.5 mins need to be educated.

Table 2. Around 2%-3% of orders exceed Upper Outlier threshold

Based on the same dataset, the Upper Outlier (74.5 mins) falls between the 97th and 98th percentile. We can calculate back that there are 2%-3% of orders with delivery time that exceed Upper Outlier threshold. Therefore, drivers with these 2%-3% of orders need to be identified and educated to improve their delivery speed.

3. Comparing with desirable impact

Objective: We want to improve customer satisfaction, therefore we want to know after how long customers start to submit complaints to CS when waiting for their orders to be delivered.

To analyze this case, we need an additional metric:

Ticket submission time (in mins) = Timestamp when CS ticket is submitted - Timestamp when order is created

For example, if an order is created at 13:00 and still not yet delivered until 13:55, then the customer creates a CS ticket to submit a complaint at 13:55, then the ticket submission time is 55 mins.

Figure 1. Density of # tickets, per ticket submission time

After plotting the ticket submission data in Figure 1, we notice that after 50 mins, the number of tickets starts to significantly increase (Disclaimer: Tickets calculated should be limited to complaints related to long delivery time only). Based on the findings, we determine 50 mins as the desired delivery time (if we deliver later than this, customers will be dissatisfied). There are around 20%-25% of orders with delivery time longer than 50 mins that we need to improve in order to maximize customer satisfaction.

Table 3. Around 20%-25% of orders may result in CS ticket submission due to customer dissatisfaction

In summary, different business objectives will bring different results of ‘late’ delivery time definition. This will also result in different strategies on how to improve the order delivery time.

Furthermore, we can also address several objectives with a combined strategy. For example, after we determine that orders with delivery time >50 mins need to be addressed to improve customer satisfaction (based on Objective #3), it is also possible to educate drivers with delivery time >50 mins as well to maximize improvement (instead of delivery time >74.5 mins, based on Objective #2).