Understanding Performance Measurement and Dashboards in Process Mining

The second lecture of the process mining course begins, focusing on performance measurement and dashboards, which were briefly mentioned in the previous week. The session aims to systematically measure process performance across various dimensions and to visualize these measures through dashboards to assist process managers and analysts in understanding and improving business processes.

The discussion highlights three main dimensions for evaluating business process performance: time, cost, and quality. The time dimension assesses whether the process is executed and completed within a timeframe that meets user needs. The cost perspective reflects the internal costs incurred by the organization to deliver the process, which is typically minimized. The quality perspective gauges customer satisfaction with the outcomes provided by the process.

The lecture introduces time measures, particularly focusing on the concept of cycle time, also referred to as lead time or throughput time. Cycle time is defined as the duration from the initiation of a process instance to its completion. For example, in an order-to-cash process, it starts when a customer sends a purchase order and ends when the delivery is completed and payment is received. The speaker emphasizes the importance of retrieving timing data from information systems, such as ERP systems, to calculate average, median, maximum, minimum, and 90th percentile cycle times, which provide insights into customer perceptions of process performance.

The speaker emphasizes the importance of contextualizing cycle time in business processes. They suggest that while customers may be more interested in the time from order receipt to product delivery, businesses should also track the time from invoice issuance to payment receipt. This distinction highlights the need to clarify which cycle time is being discussed, whether it pertains to the entire order-to-cash process or a specific sub-process.

Cycle time is defined as the total time taken to complete a process, which can be broken down into two key components: processing time and waiting time (also referred to as idle time). The speaker notes that different communities within business process management may use varying terminologies, such as 'lipton' or 'true cool time,' to describe cycle time.

To illustrate the concept of cycle time, the speaker provides a detailed example of an order-to-shipment process. They describe a scenario where a purchase order is received at 8:00 AM but is not processed until 11:00 AM due to high volume. The total cycle time from order receipt to shipment is 10 hours, but the actual processing time is only 70 minutes, consisting of 30 minutes in the morning, 30 minutes for packaging, and 10 minutes for loading. The remaining 8 hours and 50 minutes is classified as waiting time, highlighting the disparity between cycle time and processing time.

The cost of a business process is closely linked to processing time, particularly labor costs. High waiting times indicate inefficiency, prompting the use of cycle time efficiency as a performance metric. Cycle time efficiency is calculated by dividing the average processing time by the cycle time. For instance, if the average processing time for an order is 60 minutes and the cycle time is 10 hours, the cycle time efficiency would be 1 hour divided by 10 hours, resulting in a ratio of 0.1, which is considered standard for processes involving physical movement.

In practical scenarios, such as applying for a passport, the cycle time can be significantly longer than the actual processing time. For example, a passport application may take 10 days from submission to delivery, but the actual work done may only amount to 1 or 2 hours. This results in a low cycle time efficiency, highlighting the disparity between processing and waiting times in bureaucratic processes.

When discussing performance measures, it's crucial to address differences in granularity between processing time and cycle time. For example, if processing time is measured in hours and cycle time in days, conversions must be made carefully. Standard practice involves converting cycle time into working hours, considering an 8-hour workday. Thus, a cycle time of 10 days translates to approximately 64 working hours, factoring in weekends and holidays, which cannot be improved upon.

Temporal measures of performance for business processes focus on the relationship between processing time and waiting time. The cycle time efficiency, calculated as processing time divided by cycle time, is a critical metric for improvement. Modern digital-native companies are redefining expectations for cycle time efficiency, pushing beyond traditional standards where a ratio of 0.1 was considered satisfactory.

The discussion begins with an emphasis on the importance of cycle time efficiency in modern customer-oriented business processes. The speaker notes that digital native companies are pushing the boundaries of efficiency, exemplified by the quick delivery times of products, which can range from two hours to less than an hour. This efficiency is attributed to the various activities involved, such as packaging and transportation.

The speaker highlights the significance of cost measures in business processes, paralleling them with time efficiency. In business process management, costs are often analyzed on a per-instance basis, which reflects the cost incurred for each execution of a process, such as processing a single purchase order. This approach aims to ensure that customer needs are met efficiently.

A detailed breakdown of costs is presented, distinguishing between processing costs and costs of waste. The speaker explains that necessary activities for customer satisfaction are categorized as processing costs, while unnecessary activities, such as moving products within a warehouse without adding value, are classified as waste. This distinction is crucial for measuring efficiency and identifying areas for improvement.

The speaker introduces the concept of efficiency ratios, which can be calculated by comparing processing time and processing cost efficiencies. The goal is to achieve a situation where all tasks performed are necessary for serving the customer, ideally aiming for a 100% efficiency rate. This involves careful consideration of both direct material costs and resource costs associated with fulfilling customer orders.

In the context of an order to cash process, the speaker uses the example of a company selling electric appliances, such as smart TVs and vacuum cleaners. When a customer places an order online, the total cost of serving that customer comprises two main components: the material cost of the product and the costs associated with processing the order, including packing, logistics, and handling customer service issues like complaints and returns.

The speaker emphasizes the importance of distinguishing between material costs and resource costs in business process optimization. While material costs pertain to the physical items being served, resource costs focus on the human resources involved in delivering a product. The speaker notes that optimization efforts should primarily target resource costs, as material costs are better addressed through supply chain management rather than business process improvement.

A critical performance metric discussed is resource utilization, defined as the time a resource spends conducting work divided by the total available time. The speaker aims to maximize this ratio to enhance efficiency and reduce resource costs. For instance, if resources operate at 80% capacity, it leads to a decrease in the resource cost per instance of the process. However, pushing utilization beyond this threshold can lead to increased waiting times, creating bottlenecks in the workflow.

The speaker highlights the trade-off between resource utilization and waiting times. If resources are over-utilized, such as a packaging team working at 90% capacity during a high-demand period, it can result in delays and increased queues for processing orders. Effective process improvement involves managing this balance to ensure that while resources are efficiently utilized, waiting times do not escalate, which could negatively impact overall service delivery.

Quality is introduced as a third dimension of performance alongside time and cost. The speaker asserts that customer satisfaction serves as the ultimate measure of quality. Digital native companies often solicit customer feedback through star ratings after transactions, aiming to gather insights that can help improve service quality. This feedback loop is crucial for companies striving to enhance their processes and meet customer expectations.

The discussion highlights the importance of customer satisfaction metrics, specifically the Customer Feedback Score and Net Promoter Score (NPS). The NPS is calculated by subtracting the percentage of customers who would give a negative recommendation from those who would recommend the service to others. A positive NPS indicates that more customers are likely to recommend the business, which is crucial for generating additional business.

Quality measures are categorized into external and internal quality measures. External measures, such as customer satisfaction scores and NPS, are influenced indirectly by improving internal processes. Internal quality measures include defect rates, which are defined as the percentage of orders with defects compared to total orders. The speaker emphasizes that while customer satisfaction cannot be directly controlled, reducing defect rates can lead to improved customer satisfaction.

The defect rate is defined through specific examples, such as sending the wrong product to a customer or delivering to the wrong address. Each negative outcome in the process is considered a defect, and various types of defects can be measured to assess the quality of the process. The speaker notes that understanding and measuring these defects is essential for improving overall service quality.

Delivery quality measures are crucial for assessing service performance, particularly the On-Time Delivery Rate (OTDR). The OTDR is calculated by comparing the number of orders delivered within a promised timeframe (e.g., four working days) to the total number of orders. A high OTDR, such as 90%, indicates that the business is meeting its delivery commitments effectively.

In addition to the On-Time Delivery Rate, the variance of cycle time is another important performance measure. The speaker suggests calculating the variance of delivery times to understand the consistency of the delivery process. This statistical measure helps identify fluctuations in delivery times, which can impact customer satisfaction and operational efficiency.

Cycle time follows a normal distribution, and the standard deviation is used to characterize variance. A smaller standard deviation is preferred to ensure predictability in service delivery. For instance, if the Estonian passport office promises delivery within 10 working days, variability in actual delivery times can lead to customer dissatisfaction. If one customer receives their passport in 2 days while another waits 5 days, it creates confusion and anxiety. Therefore, maintaining a consistent delivery time, such as 8 days with a variance of 1 day, enhances customer trust and satisfaction.

Three key quality measures to monitor in business processes are on-time delivery rate, cycle time variance, and defect rate. These metrics are crucial as they directly impact customer perception of service quality. Customers value timely delivery, consistent quality, and reasonable costs, which ultimately influence their satisfaction and loyalty.

The main dimensions of performance that matter to customers are time, quality, and cost. These factors not only affect customer satisfaction but also determine pricing strategies for companies. Understanding these dimensions is essential for improving business processes and meeting customer expectations.

The case arrival rate is a measure of demand for a process, defined as the number of new cases created per time unit, such as orders per hour. For example, an online company selling electronic appliances may receive an average of 10 orders per hour, which is denoted by the Greek letter lambda (λ). This metric helps businesses gauge demand and adjust their operations accordingly.

Inter-case arrival time, the time between two arrivals, is inversely related to the case arrival rate. For instance, if the arrival rate is 10 orders per hour, the average time between orders is 6 minutes. Understanding both the arrival rate and inter-arrival time is crucial for managing workflow and meeting customer demand effectively.

The speaker discusses the concept of 'Work in Progress' (WIP), which quantifies the active cases at any given time. For instance, an online shop with a WIP of 100 indicates that there are currently 100 orders received but not yet delivered. In an IT helpdesk scenario, if 50 requests are received per hour and there are 20 staff members, a WIP of 150 requests signifies the number of unresolved customer complaints.

The speaker introduces Little's Law, which establishes a relationship between WIP, the arrival rate of cases (lambda), and cycle time. According to this law, WIP equals the case arrival rate multiplied by the cycle time. For example, if the arrival rate is 10 customers per hour and the cycle time is 2 hours, the average number of open claims would be 20 at any given time.

The speaker explains the practical application of Little's Law in determining cycle time when direct measurement is challenging. In a restaurant scenario, if 50 customers enter per hour and the average number of customers being served at any time is 200, the cycle time can be calculated as 200 divided by 50, resulting in an average customer stay of 4 hours. This illustrates how WIP and arrival rates can help infer cycle times in various contexts.

The speaker discusses the impact of COVID-19 on restaurant capacity, illustrating that if a restaurant has a capacity of 100, it must reduce this to 50 due to health regulations. To manage this change, the speaker references the Little's Law, explaining that to decrease the average number of customers (weight) in the restaurant, one can either reduce the arrival rate (lambda) or decrease the cycle time, which involves ensuring customers spend less time in the restaurant.

The speaker introduces various performance measures relevant to process management, including time, cost, quality, demand, and workload measures. They emphasize the importance of identifying which performance measures to focus on by employing a top-down approach, starting with the organization's objectives and areas needing improvement.

The speaker explains the Balanced Scorecard method used by organizations to define objectives and key performance indicators (KPIs) for the year, such as for 2021. These KPIs are categorized into four dimensions: financial, customer, internal process, and innovation and learning. This framework helps managers at various levels understand and communicate performance measures effectively.

The discussion highlights the issue of customer churn, with the speaker noting that a company may lose 10% of its customers in a year. This loss prompts upper management to set a target to improve customer loyalty by 10%. The process manager is tasked with analyzing the order-to-cash process to identify reasons for customer attrition, potentially through surveys or data analysis, leading to the conclusion that timely service is a critical factor.

The speaker emphasizes the importance of timely service in the restaurant industry, suggesting that customers expect to be served in less than 30 minutes. As a process manager, one should focus on measuring the percentage of customers served within this timeframe to ensure satisfaction and improve retention.

The speaker emphasizes the importance of the on-time delivery rate, aiming for it to be as close to 100% as possible. Currently, 80% of customers are served on time, while 20% are late. The goal is to increase the on-time service rate to 90% by the end of 2021, which will trigger various process improvement initiatives.

The speaker discusses the existence of comprehensive performance measurement frameworks, such as the Score Reference Model, APQC Process Classification Framework, and IT Infrastructure Library. These resources provide extensive definitions of processes and associated performance measures, which can guide organizations in identifying relevant temporal, cost, and quality measures for their specific processes.

Business process performance monitoring is introduced as a critical set of activities within business process management. It involves monitoring processes using data from enterprise systems, which are packaged into event streams or logs. The speaker highlights two main categories of techniques: performance dashboards and process mining, with the former focusing on visualizing performance and the latter on deeper analysis of performance issues.

The speaker outlines the importance of designing performance dashboards tailored to the specific stakeholders who will use them. It is crucial to identify whether the dashboard will serve operational, tactical, or strategic purposes, as this decision simplifies discussions among the design team and ensures the dashboard effectively meets user needs.

An operational dashboard is designed for workers involved in processes, such as those in packaging at an online shop selling electronic appliances. It provides real-time or near-real-time data to help them manage their daily tasks, such as the number of orders to package. Key performance measures displayed include the working process, categories of work, problematic cases at risk of delays, and resource load, indicating how busy different teams and robots are in the warehouse.

Tactical dashboards cater to process owners and managers aiming to enhance processes. They focus on performance comparisons, such as analyzing January's performance this year against last year. Managers seek insights into cost sources, bottlenecks, error rates, and resource utilization. These dashboards help in decision-making for process improvements, such as identifying areas for enhancement by September of the current year. They may display cycle time distributions and performance metrics across different geographic regions.

Strategic dashboards are intended for upper management and executive managers, addressing high-level questions regarding customer loyalty, cost management, and sustainability targets like CO2 emissions. These dashboards are structured around key performance areas such as customer satisfaction and feedback, allowing executives to assess performance across various process groups, including repair and delivery processes. Although not the focus of the current course, understanding their existence is crucial.

The discussion begins with the importance of defining performance areas, such as high-level process performance, financial performance, and customer excellence performance. The speaker emphasizes the need to refine these areas hierarchically until reaching measurable performance metrics that can be derived from underlying systems.

A distinction is made between operational tactical dashboards and strategic dashboards. The speaker clarifies that this course will focus primarily on tactical dashboards, with some attention to operational dashboards, while strategic dashboards will be covered in courses related to strategic management.

The speaker introduces a five-step method for designing a tactical dashboard. This process begins with identifying the target user and understanding their questions and needs. Based on this information, the necessary dashboard elements are determined to effectively answer these questions.

The speaker outlines five types of elements commonly found in dashboards: indicators (rectangles displaying performance measures), trend charts (showing the evolution of performance measures over time), performance distribution charts (illustrating performance across different sections), and cross-sectional shots (displaying performance measures across defined dimensions such as geography or business units).

The speaker explains performance distribution charts, which can illustrate the distribution of cycle times in a process. For example, a histogram may show that most cases take around five days, indicating a healthy process with few outliers taking significantly longer.

Cross-sectional shots are described as performance measures displayed for each group defined by a chosen dimension, such as location or product type. The speaker provides an example of displaying performance measures for each state in a country, highlighting the importance of this analysis in understanding performance across different segments.

The speaker mentions various tools available for creating process performance dashboards from execution data, specifically highlighting a category known as business activity monitoring. These tools are noted for their effectiveness in building dashboards that reflect real-time performance data.

The discussion begins with an overview of operational dashboards, highlighting that Oracle and SAP provide specific tools for performance management. Various business intelligence tools, such as Microsoft Power BI, ClickView, and Tableau, are commonly used in companies to create tactical dashboards. Additionally, process mining tools, which will be introduced in the course, also feature dashboard modules for tactical dashboard creation. The speaker notes that strategic-level dashboards, often referred to as balanced scorecards, will not be covered in this course.

The speaker transitions to demonstrating a specific tool called Promora, which will be introduced in the following week. This tool allows users to connect to enterprise systems or manually upload data. An example dataset from a repair process is presented, illustrating how individuals report issues with devices like laptops or audio-visual equipment. The dataset spans three months and contains historical data relevant to the repair process.

Using the Promora tool, the speaker showcases various dashboard features that provide insights for tactical managers and process owners. Key indicators displayed include the number of cases during the specified time period, event counts, and case duration, which is synonymous with cycle time. The dashboard also visualizes the distribution of active cases over time, identifying peaks in workload, such as higher case counts on January 12th compared to January 5th and January 19th.

The speaker explains different data visualization techniques used in the dashboard, including trend charts, which illustrate case distribution over time, and longitudinal charts that depict case duration distributions. The data appears to follow a normal distribution. Additionally, cross-sectional charts are utilized to visualize performance across various activities within the process, allowing for a comparative analysis of time taken for different tasks.

The discussion concludes with an emphasis on the customization capabilities of tactical dashboarding tools like Promora. Users can create their own charts, add new indicators, and select specific data to display, such as the distribution of cases by defect type or the number of cases handled by different testers. The speaker indicates that this topic of customizing performance dashboards will be revisited in lesson five, focusing on performance mining and the analysis of process performance using process mining tools.