Understanding Automated Process Discovery and BPMN in Process Mining

This lecture marks the fourth session of the course, focusing on the capabilities of process mining. The instructor outlines that there are four main capabilities to be covered: automated process discovery, conformance checking, performance mining, and analysis of process variance. The course is currently at its midpoint, with a homework assignment to be released today, due in two weeks, and discussed in the practice session scheduled for tomorrow.

The primary focus of today's lecture is on automated process discovery. The instructor explains that given an event log with minimum requirements, it is possible to explore the corresponding process through visualizations, specifically using process maps or models. This exploration is crucial for understanding the underlying processes represented in the event logs.

The basic visualization provided by process mining is the Directly Follows Graph (DFG), also referred to as a process map. This graph consists of nodes representing activities from the event log and edges indicating directly follows relations, where one activity occurs immediately after another. The start node is marked with a play button, while the end node is indicated with a stop button, highlighting the flow of activities within the process.

In the DFG, there are special arcs that denote significant relationships. An arc from the start event to an activity indicates that the activity can be the first in the process, while an arc from an activity to the end node signifies that the activity can be the final one in its trace. These distinctions are essential for identifying incomplete cases where processes may be prematurely interrupted.

The instructor emphasizes the importance of arcs that loop back to the same activity, indicating direct rework within the process. Each activity and arc in the DFG is typically associated with performance statistics, such as case frequency, which reflects how many times an activity or directly follows relation occurs. For instance, the activity 'register' was performed 1,104 times, indicating its frequency and the subsequent activity 'analyze defect' followed in the same number of cases.

The discussion introduces performance mining, emphasizing the ability to overlay or color code nodes in a process map using various statistics. Key metrics include maximum frequency of activity occurrence, minimum frequency of activity occurrence, and the duration of activities, which encompasses processing and waiting times. Different tools offer diverse options for overlaying performance statistics on process maps, which are central to the field of process mining.

To construct a process map from an event log, the first step involves extracting distinct traces of executions from the log. The speaker illustrates this by simplifying activity names to single letters (a, b, c, etc.) and sorting events in each case by their timestamps. The sequence of activities observed in a case is then recorded, with an example given where five cases follow the same pathway of activities (a, b, c, g, e, h). Each distinct trace, or case variant, is assigned a count reflecting its occurrences in the log.

The speaker explains the concept of case variant abstraction in process mining, where an event log is represented as a collection of case variants. In the example, five case variants are identified, with varying frequencies of occurrence: the first variant appears five times, the second four times, the third three times, and the fourth three times. This abstraction simplifies the log by grouping activities and counting their occurrences.

To create a process map, the speaker describes traversing the identified case variants from left to right, representing the first trace variant as a linear graph. Each direct follow relation between activities is recorded, with the frequency of occurrences noted on the arcs. For instance, the first trace variant occurs five times, leading to a notation of '5' on each arc. As the speaker moves to the second trace variant, they update the frequencies based on shared activities, illustrating how new activities can alter the path.

The speaker discusses the 'directly follows' relation in a process map, noting that they have observed this relation only four times so far. They illustrate this by drawing a line from node C to F, indicating the direct follow. The speaker continues to trace the process, moving from F to G, and then from G to H, marking the last activity in this trace variant. They annotate the relation with a frequency of four, as it was not seen in the first trace.

In the analysis of the third trace case variant, which starts with node B, the speaker adds a frequency of three to the 'directly follows' relation from A to B, as it has occurred three times. They introduce a new node D, drawing a direct follow from B to D, and note that this transition has been observed three times. The speaker continues tracing from D to G, then G to E, and finally E to H, where they have seen the relation five times before, adding three more to the frequency.

The speaker moves to the final trace, starting from A to B, adding three to the frequency of this direct follow relation. They continue from B to D, which has been observed before, adding three to its frequency. The transition from D to E is new, so they create this relation with a frequency of three. The tracing continues from E to G and from G to H, where they again add three to the frequency of the direct follow relation, as it has been seen in the second trace variant.

The speaker describes a simple technique for mapping the process, where they scan the collection of trace variants event by event. They meticulously record how many times each 'directly follows' relation has been observed, ensuring a comprehensive understanding of the process flow.

An exercise is introduced where the speaker invites participants to draw the process map based on a provided event log. The log is displayed in two columns due to slide constraints, starting with case ID one and moving to a reminder, then transitioning to the second column with a reminder and closing the case. The speaker emphasizes the importance of reading the log correctly across both columns.

The speaker lists the trace variants as they prepare to draw the process map. The variants include sequences such as 'ff sb', 'a r ppcc', and 'if f is b r r r ppcc'. They note the execution of the second case, which occurs only once, indicating a frequency of one for each step in this case.

The speaker discusses the construction of directly follows relations in a sequence of activities. Starting from 'ff' to 'sb', a plus one is added, and then from 'sb' to 'r', a new directly follows relation is established. The speaker notes that this case variant occurs once, so they record a count of one. The process continues with 'r' moving to 'pp', another new relation, and a count of one is noted again. The transition from 'pp' to 'cc' is also recorded, resulting in a total of two for that segment.

The speaker elaborates on counting the occurrences of transitions between activities. They mention going from 'r' to 'r' multiple times, indicating a direct follow relation from 'r' to 'r' for the first time, which is recorded as one. The speaker continues to track the transitions, noting that they move from 'r' to 'r' again, leading to a count of two. This meticulous counting continues as they traverse through 'pp' to 'cc', culminating in a total of four for that sequence.

The speaker highlights the potential of using the same approach to calculate timestamps for activities, allowing for the determination of start and end times. They express that this method can facilitate the creation of a process map, which serves as a simple yet effective visualization of business processes. The speaker emphasizes the ability to derive various statistics, such as the duration of activities, by replaying the log and updating directly follows relations.

In summarizing the construction of process maps, the speaker points out significant limitations inherent in these visualizations. They caution that process maps can create illusions, particularly regarding the existence of certain paths or cycles that may not actually occur. The speaker illustrates this with an example, noting that while a process map may suggest a loop between activities 'g' and 'e', the original event log does not support the notion of these activities being repeatable within the same case.

The speaker discusses the limitations of process maps, highlighting that they can create an illusion of repetition even when there is none. Using a simple event log example with traces 'a b c d' and 'a c b d', the speaker illustrates how the process map suggests a loop between activities 'b' and 'c', despite 'b' only occurring once in each trace. This confusion arises from the nature of directly follows graphs, which can misinterpret parallelism as repetition.

To address the limitations of process maps, the speaker introduces BPMN (Business Process Model and Notation) as a more sophisticated approach. BPMN allows for the creation of detailed process models that can accurately represent business processes. These models are essential for business analysts to document processes, communicate with colleagues, and identify areas for improvement. The speaker notes that many organizations, including insurance companies and government agencies, maintain extensive repositories of process models to manage their operations effectively.

The speaker explains the fundamental components of a BPMN model, which consists of four types of elements: rectangles representing activities, events marking the start and end of processes, and gateways that dictate the flow of activities. The speaker emphasizes that while BPMN is a complex language with over a hundred symbols, understanding the basic structure is sufficient for the course's focus on automatically discovering process models. The discussion includes the importance of events and gateways in determining the execution order of activities.

The discussion begins with the introduction of key elements in process modeling, specifically activities, events, and gateways, which are interconnected through sequence flows. These flows illustrate the direct relationships between activities, such as moving from one activity to another or arriving at a gateway that necessitates branching.

The speaker elaborates on exclusive gateways, which are crucial for capturing decision points in a process. An exclusive gateway, also known as an XOR split, allows for a choice between two paths based on certain conditions. It has one incoming flow and two outgoing flows, indicating that a decision must be made at this juncture, leading to either the left or right path depending on the evaluated condition.

Following the discussion on exclusive gateways, the speaker introduces exhaustive joint gateways, which are used to merge branches back into a single flow. This type of gateway has two incoming arcs and one outgoing arc, allowing for the continuation of the process regardless of the path taken to reach it. The speaker provides an example involving invoice processing, where the verification of an invoice leads to different outcomes based on the results of that verification.

In the context of invoice processing, the speaker outlines three potential outcomes after verifying an invoice. The first outcome is to post the invoice for payment. The second involves addressing mismatches, which can either be corrected by resending the invoice to the customer or blocking the invoice if it is identified as a duplicate. Regardless of the path taken, the final activity in the process is to park the invoice, marking the completion of the process.

The speaker transitions to discussing parallel gateways, also referred to as AND gateways. These gateways indicate that when a certain point in the process is reached, two branches can be executed simultaneously. Alternatively, they can also signify that the process will wait for all branches to complete before proceeding to the next activity. The speaker emphasizes that parallel gateways, like exclusive gateways, can also take on two forms, where a split gateway generates multiple tokens to denote parallel execution.

The parallel joint gateway is designed to receive tokens from multiple incoming branches, producing a single token in the outgoing branch once all branches have completed their processes. This synchronization ensures that all parallel activities are completed before proceeding, as illustrated by the example of an airport security check where both the passenger and their bag must be screened simultaneously.

The AND gateway operates by producing two outgoing tokens upon receiving an incoming token from one of its branches. This allows for parallel processing of activities, such as personal security screening and luggage screening at an airport. Both tokens must arrive at the end join for the process to continue, demonstrating the necessity of synchronization in multi-branch workflows.

In a business scenario involving purchase orders from various customers, the company manages two warehouses located in Amsterdam and Hamburg, each storing different products. When a purchase order arrives, it is classified based on its line items, determining whether to send it to one warehouse or both. This process involves decision-making to ensure that orders are fulfilled efficiently, with the possibility of synchronizing deliveries from both locations.

A more streamlined approach to handling orders involves checking for products from each warehouse in parallel. If products from Hamburg are present, the order is sent there; if products from Amsterdam are included, it is sent to Amsterdam. This method allows for flexibility, as it accommodates orders that may not contain items from either warehouse, thereby optimizing the order fulfillment process.

The speaker explains the concept of an exhaust split gateway, which can be connected directly via an arc to an XOR joint gateway. This setup allows for the capture of scenarios where a branch is entirely skipped, as demonstrated in the provided diagram. The speaker notes that both at the top and bottom of the diagram, there are instances of skipped branches, where one task is omitted after processing in both warehouses. The synchronization occurs at the AND gateway, leading to the registration of the order, which is then marked as ready for shipment.

The discussion shifts to OR gateways, which come in two types: split and join. The speaker describes the functionality of a north split gateway, which allows for the initiation of one or more branches based on certain conditions. The branches can converge at an OR join, which intelligently waits for the necessary branches to complete before proceeding. The speaker illustrates this with an example where the order's arrival triggers a check on the line item, determining whether to proceed with one or both branches, and how the OR join gateway manages synchronization.

The speaker emphasizes the importance of selecting the appropriate gateway type for different scenarios. They advocate for using specific gateways like AND, XOR, or OR based on the semantics required for clarity in communication. The choice of gateway impacts the understanding of whether a point is for synchronization or merging of branches. The speaker aims to provide the most informative gateway type to enhance user comprehension.

In a practical exercise, the speaker presents a process model featuring activities labeled A through F and highlights that it can have two end events. They explain how to convert this model into one with a single end event by drawing an arc from task E to the end event following task F. The speaker notes that having two end gateways is a matter of convenience and can be merged into a single joint node.

The speaker poses a question regarding the specification of a gateway marked with a question mark in the process model. They inquire whether to use an AND, XOR, OR, or even a PLUS gateway. The speaker critiques the PLUS gateway, describing it as ineffective because it would always wait for tokens from both branches, potentially leading to a deadlock if one branch encounters an exception. This highlights the importance of choosing the right gateway to avoid process stagnation.

The speaker discusses the execution of task 'f' in a process flow, emphasizing that it will not start if the second token is not received. This highlights the importance of token reception in determining task execution order, particularly after task 'c'. The speaker notes that using an XOR gateway would lead to unintended double execution of task 'f', which is not the intended outcome.

The speaker explains the advantages of using an OR gateway over an XOR gateway in process modeling. The OR gateway allows for the execution of task 'f' when tokens are received from either branch, ensuring that task 'f' is executed as expected without duplication. This choice is crucial for accurately capturing the intended process flow.

The speaker introduces the concept of repetition in process flows, particularly in the context of preparing and reviewing responses. This is represented by an XOR gateway that allows for a decision point where the process can either continue or revert to repeat previous activities. The repetition can occur multiple times until a specified condition is met, demonstrating the flexibility of process design.

The speaker mentions algorithms that can generate BPMN diagrams from process maps and event logs, aiming to capture various pathways observed in the logs. However, they caution that real-life business processes are often messy, leading to complex diagrams that resemble 'spaghetti'.

To manage the complexity of process diagrams, the speaker discusses two strategies: abstraction and filtering. Abstraction involves hiding certain arcs or nodes to focus on the most frequent or significant pathways, while filtering entails preprocessing event logs to retain only a subset of cases based on specific criteria. This results in simpler process maps that are easier to analyze.

The speaker provides an example from a real-life hospital event log involving approximately 1,050 cases of patients treated for sepsis. The complexity of the process map derived from these cases illustrates the challenges faced in understanding and analyzing real-world data, emphasizing the need for effective abstraction and filtering techniques.

In the context of incident management at a bank, there are approximately 50,000 cases, which is considered a manageable number. However, the corresponding process map for incident management is deemed ineffective for visual interactive exploration, which is the primary goal of automated process discovery. The speaker emphasizes the need for abstraction and filtering, and suggests switching to the BPMN perspective to create a more readable and insightful representation of the process.

The discussion transitions to a production process within a manufacturing execution system (MES), which is utilized in factories to record production batches. The speaker describes various stations on a factory line that perform mechanical operations such as turning, milling, nitration, and wire cutting to produce a final metallic product. The complexity of the full process map is highlighted, indicating that it becomes difficult to comprehend without the use of abstraction tools.

To manage the complexity of the process map, the speaker demonstrates the use of abstraction sliders to filter out arcs with the highest frequency, retaining those with lower frequencies. This method allows for a clearer view of the process, although some areas remain cluttered, raising questions about the presence of loops versus parallelism in the process. The speaker notes that process maps often struggle to differentiate between these two aspects.

By switching to the BPMN view, the speaker is able to visualize gateways and identify repetitions within the production process. This view clarifies that certain activities, such as turning and milling, may be repeated rather than indicating true parallelism. The speaker adjusts the parallelism slider to explore the presence of parallel processes, noting that production processes tend to be predominantly serial, with only occasional instances of parallelism.

The speaker shifts focus to a hospital patient treatment process, attempting to visualize it at different abstraction levels. At 50% visibility, the process appears messy, with short loops identified between activities. The speaker continues to adjust the visibility settings to find a balance that allows for a more readable representation while still capturing the complexity of the process.

The speaker discusses the distinction between repetition and parallelism in activities, utilizing a BPMN (Business Process Model and Notation) view to illustrate the process. They note that there are decision points where they may choose to administer antibiotics or conduct tests like CRP (C-reactive protein) and leukocyte tests, which can be repeated multiple times. The speaker emphasizes the complexity of the process, where branches of activities converge, leading to patient outcomes such as lactic acid tests, IV liquid administration, and eventual patient release, sometimes resulting in return visits to the emergency room.

The speaker highlights the usefulness of both BPMN and process map views for understanding processes. They mention that while BPMN provides detailed insights, it can be challenging to interpret due to the complexity of splits and joins. The speaker prefers to start with a process map view to identify messy parts before switching to BPMN for a complementary perspective, indicating that both visualizations serve distinct purposes in process analysis.

The speaker explains that abstraction in process visualization can obscure details, as it involves hiding certain direct relationships within the process. They advocate for combining abstraction with filtering, which allows for the removal of specific cases from logs to focus on relevant data. Various filters are introduced, including case variant filters for frequent pathways, case ID filters for specific patients, and attribute filters for activities performed by certain individuals. The speaker emphasizes the importance of distinguishing between case filters, which remove entire cases, and event filters, which selectively remove events within cases.

The speaker elaborates on the concept of event filtering, which allows for the removal of uninteresting events, such as those performed automatically by a system, while retaining the overall case. This selective filtering is crucial for focusing analysis on manually performed activities. The speaker acknowledges the complexity of managing event logs and provides a rough guideline for beginners: open the log, adjust the abstraction slider, switch to the BPMN view, and apply relevant filters to streamline the analysis process.

The speaker discusses various techniques for analyzing event logs, emphasizing the importance of switching perspectives to gain insights from different angles. This includes not only examining activities but also understanding the overall structure of the process, identifying key elements such as parallelism, branching points, and instances of rework. The goal is to capture a comprehensive view of the process for effective analysis.

An automated process discovery analysis template is introduced, consisting of a single table divided into four sections. The left side outlines the types of analyses one might conduct on an event log, while the right side provides guidance on how to perform these analyses. This structured approach aids in systematically exploring event logs.

The first step in exploring an event log using process mining is flow analysis. This involves examining the structure of the process, identifying main case variances, and analyzing whether cases finish earlier than expected or remain incomplete. The speaker highlights the significance of understanding these dynamics to improve process efficiency.

After completing flow analysis, the next step involves applying filters to examine different components of the process separately. For instance, one might analyze cases of patients returning from emergency rooms versus those who do not, or compare outcomes based on age groups, such as patients over and under 60 years old. This targeted analysis helps in understanding specific trends and behaviors within the process.

The speaker emphasizes the importance of analyzing the frequency of activities within the process. This includes identifying the most commonly performed activities and the pathways frequently followed. The analysis should focus on total activity frequency rather than just average frequency, providing a clearer picture of process dynamics.

An interesting approach discussed is switching perspectives during analysis. Instead of solely viewing the process from the activity standpoint, one can also analyze it from the workers' perspective, examining how tasks are handed off between resources. This perspective can yield valuable insights into the efficiency and effectiveness of the process.

The speaker mentions the use of process mining tools, such as Promore, to facilitate flow analysis and filtering. The following slides in the presentation will provide detailed steps for conducting flow analysis, including identifying key filters and analyzing frequent pathways and resource handoffs within the process.

The discussion concludes with a note that not all elements of the analysis template will be covered in detail. Instead, the focus will be on selectively exploring specific aspects of business processes through practical exercises, allowing for a deeper understanding of the methodologies discussed.