Generating Privacy-Preserving Process Data with Deep

Generative Models
Keyi Li1, Sen Yang2, Travis M. Sullivan3, Randall S. Burd3, Ivan Marsic1
1Electrical and Computer Engineering Department, Rutgers University, 2LinkedIn, 3 Children’s National Hospital
{kl734, sy358, marsic}@[Link], {tsullivan, rburd}@[Link]

ABSTRACT lack of data is a barrier of process mining studies in many do-

mains. This work addresses the challenge of sharing process data
Process data with confidential information cannot be shared di-
with a privacy-preserving requirement. The principle is to first
rectly in public, which hinders the research in process data min-
learn the underlying data distribution by learning representative
ing and analytics. Data encryption methods have been studied to
deep sequential models (e.g., gated recurrent unit (GRU) [5], long
protect the data, but they still may be decrypted, which leads to
short-term memory networks (LSTM) [10], Transformer [26]),
individual identification. We experimented with different models
and to use these models to generate high quality synthetic pro-
of representation learning and used the learned model to gener-
cess data (Figure 1). The synthetic data generated will have a
ate synthetic process data. We introduced an adversarial genera-
similar distribution as actual data and can be shared without
tive network for process data generation (ProcessGAN) with two
compromising confidentiality. In addition, the size of the process
Transformer networks for the generator and the discriminator.
data can be increased to include novel process sequences.
We evaluated ProcessGAN and traditional models on six real-
Process data is sequential data constructed from a series of
world datasets, of which two are public and four are collected in
activities. Process data varies according to use cases. Simple pro-
medical domains. We used statistical metrics and supervised
cesses can be almost linear (i.e., single path) with only a few ac-
learning scores to evaluate the synthetic data. We also used pro-
tivities, while complex processes can have parallel paths, loops,
cess mining to discover workflows for the authentic and synthet-
intermediate dependencies, and hundreds of different activities.
ic datasets and had medical experts evaluate the clinical applica-
The recurrent neural networks (RNN), such as GRU and
bility of the synthetic workflows. We found that ProcessGAN
LSTM, have been successfully applied to learning sequential data
outperformed traditional sequential models when trained on
[21]. For long and complex sequences, the RNNs may have
small authentic datasets of complex processes. ProcessGAN
drawbacks. First, the inner dependencies of the tokens in these
better represented the long-range dependencies between the ac-
sequences are difficult to capture by the RNNs. Second, when the
tivities, which is important for complicated processes such as the
training data size is small, convergence of the RNNs can be diffi-
medical processes. Traditional sequential models performed
cult. These factors may impact the reliability of the synthetic
better when trained on large data of simple processes. We con-
process data generated by RNNs.
clude that ProcessGAN can generate a large amount of sharable
synthetic process data indistinguishable from authentic data. Representation
Learning Generative Model: Generate
Authentic RNN, LSTM, GAN, Synthetic

CCS CONCEPTS Process Data Transformer, … Process Data

• Information systems → Data mining; • Computing

methodologies → Artificial intelligence Random Initial
Tokens

KEYWORDS Figure 1: Our framework of generating synthetic process

data by representation learning from authentic process
Synthetic data generation; Process mining; Sequential data; Gen-
data.
erative adversarial networks; Data privacy
We aim to explore different methods for synthetic sequence
1 Introduction generation that address different degrees of complexity. We fo-
Process mining (PM) techniques have been applied to discover cused on (1) generating plausible synthetic process data with few
knowledge about processes in different disciplines, including process errors, (2) learning the authentic data distribution when
healthcare [20], business [25], and education [3]. Process data in data size is small, (3) generating synthetic process data with only
these fields usually contains confidential information, adding a random initial tokens as inputs to preserve the privacy of au-
requirement for addressing privacy concerns and obstacles relat- thentic data, and (4) generating new process sequences that are
ed to data sharing [23]. Public process datasets are limited com- plausible based on the actual process. Inspired by the image gen-
pared to other data types such as images, text, and sensor. Data eration with random noises, we applied the Generative Adver-
sharing has led to advancement in many research fields (e.g., sarial Networks (GANs) for process data generation [6]. We pro-
ImageNet in computer vision and GLUE in natural language pro- pose ProcessGAN, built based on the encoder part of Transform-
cessing) by standardizing the research practice and enabling a
focus on model exploration and benchmark comparisons. The
er and a GAN structure, to generate process data1. The Trans- previous generated steps) of the Transformer is not computa-
former model uses the multi-head self-attention mechanism to tional efficient. We sought to address these challenges by devel-
address long-term dependencies, while the GAN structure in- oping a more efficient non-autoregressive training (i.e., generat-
creases the precision of the synthetic sequences when data size ing all tokens in a sequence in parallel) of the Transformer and
is insufficient. Unlike the RNNs that take one initial token as in- adding the GAN structure for better generation performance.
put to generate a sequence step-by-step, the Transformer model
can take a full random sequence as input. The ProcessGAN mod- 2.2 Generative Adversarial Networks
el can then expand the exploration space and generate more var- Our idea of generating sharable synthetic data was inspired by
iable process sequences. the concept of image data augmentation in machine learning
We evaluated the synthetic data quantitatively by measuring tasks [22]. GAN-based neural networks have been trained on
the underlying distributions and qualitatively by assessing the limited authentic data to enhance data size while capturing the
semantics. We calculated basic statistical measures between the features and generating plausible images using only random
synthetic data and the authentic data to compare the perfor- noises [6]. When using GAN, a discriminative network 𝐷 was
mances of different models. We used both a data-driven method trained to distinguish between authentic and generated synthetic
and a knowledge-based assessment to evaluate the semantics. data, while a generative network 𝐺 was trained to generate syn-
We did a case study by inviting domain experts to check the
thetic data from random input and tricks 𝐷. 𝐺 gradually learned
process errors occurred in the synthetic process dataset.
to mimic the features of the authentic data and learned to gener-
The main contributions of this paper are:
ate synthetic data that is similar to actual data. This high-quality
• A framework for generating privacy-preserving process data. synthetic data can be used to augment training dataset for ma-
We compared different models and analyzed their merits. chine learning and can be shared preserving privacy.
• A novel GAN-based Transformer network to generate realis- GAN-structured generative networks have also been applied
tic process data. In this model, we leveraged Gumbel-Softmax for text data generation [9, 17, 30-32]. In these works, generators
to handle the discrete backpropagation from the discriminator are usually RNN-based models, and discriminators are usually
and added activity distribution into the loss function to Convolutional Neural Network (CNN)-based or RNN-based
achieve better generation performance. models. Because the gradients from the discriminator cannot
• A solution for evaluating the synthetic process data. The solu- back-propagate through discrete variables, SeqGAN was intro-
tion includes three different dimensions: (1) statistical mea- duced that use reinforcement learning (RL): each time a new to-
sures, (2) supervised learning score, and (3) process mining. ken is generated, 𝐺 simulates the whole sequence and lets 𝐷
score the sequence, then 𝐷 gives an RL reward to that token [11,
2 Related Work 30]. With the RL reward given by 𝐷, some error tokens will have
lower scores and will be less likely to be generated. 𝐺 then learns
2.1 Process Data Generation the intermediate dependencies of the tokens and generates high-
Process data can be difficult to collect and share because of com- quality sequences.
plexity and inclusion of private information [27]. The hidden Although the RL rewards given by the GAN discriminator
Markov model has been used to augment process datasets to can reduce some errors, the training of these GANs is difficult.
capture the sequential dependencies of activities and generate The discriminator's high variance of RL rewards may confuse the
synthetic process data [29]. In recent years, some deep genera- generator. The training process may experience mode collapse
tive models, such as the RNN-related models, Transformer mod- because the generator cannot learn a stable representation from
el, and GANs, were applied to the process mining field to do the the discriminator [14]. These problems are obstacles for sequen-
process events prediction task [2, 16, 24]. These models can also tial data generation, especially for complex process data.
be used to generate process data. The difference is that the pre-
diction task usually applies the teacher forcing mechanism to do 3 Method
the training and prediction (i.e., using the activities in the
In this section, we introduce the ProcessGAN model and meth-
ground truth sequences to predict the next activity). However, in
ods for evaluating the synthetic process traces (“sequences” and
the data generation task, we do not have access to the ground
“traces” are used alternatively in the following sections). In Pro-
truth sequences, and the next token is generated based on the
cessGAN (Figure 2), we use the Transformers (encoder only) as
previous generated tokens. In the case of some long complex
both the generator 𝐺 and discriminator 𝐷 of the GAN architec-
processes, RNNs may suffer exposure bias [1, 19], i.e., errors oc-
ture. The input to the generator is a random sequence, and the
curring in the generated tokens propagate during data genera-
output from the generator is a sequence of vectors. Each dimen-
tion. These errors may reduce the authenticity of synthetic pro-
sion of a vector has each activity’s probability. We then extract-
cess data. The Transformer network can manage long-term de-
ed the most likely activities from the vectors and built a synthet-
pendencies using the self-attention mechanism, but the auto-
ic activity sequence. Considering that this extraction is non-
regressive training (i.e., generating the next step based on the
differentiable, we applied straight-through Gumbel-Softmax to
1 The source code is available at [Link]
obtain the gradients to update the parameters of the generator.
The synthetic and authentic process sequences are then fed into
 *(()

Random Input -1
Gumbel
Sequences (
Softmax -2
Batch Size
" -3
Embedding Vocabulary
-4
… Size $!"# Generator Size $$
-5
Activity
Embedding …
Sequence -&'
Length !
Add up same type of activity
%&''%
!"#(%(&), )) Discriminator
%&''%&
Activity
*+,,!
Divergence
Add up same type of activity

Authentic -1
Batch Size Process
-2
Sequences )
"
-3
Vocabulary
-4
… Size $$
One-hot -5
Encoding …
Sequence
Length ! -&'

Figure 2: The architecture of the ProcessGAN network. The random input sequences and authentic process data (au-
thentic activity sequences) are preprocessed to the same length. Discriminator is trained to distinguish the authentic
and synthetic sequences. Generator is optimized by the losses given by the discriminator and the activity divergence.

the discriminator for adversarial learning. We included an auxil- Our experiment shows the end token in the random sequence
iary loss that considers the activity distribution divergence to will not affect the result.
guide adversarial learning. We propose several evaluation ap- Generator. The generator 𝐺 takes random sequences 𝑍 as input.
proaches of different perspectives, like data statistics, supervised We first embed the activities in 𝑍 (like word-embedding in NLP).
learning, and process mining. We set the embedding size (𝑁'() ) according to a rule of thumb:
𝑁'() = !4𝑁" (in NLP the rule of thumb is "4𝑁" . For process da-
3.1 ProcessGAN Model ta, the activity vocabulary size is usually smaller, so a good start
Transformer and GAN. Traditional Maximal Likelihood Esti- point is !4𝑁" ).
mation (MLE) training of the RNN-related networks suffers from The Transformer network uses the self-attention mechanism
exposure bias, i.e., RNNs tend to overfit the exposed data and to learn each token’s position and correlations with other to-
would not explore the vast output space beyond the exposed da- kens. The rich dependencies between the activities in a process
ta distribution [19]. If a sequence generated from an unobserved (i.e., one activity could have multiple relations with other activi-
starting token, divergence may occur and become enlarged along ties) were captured based on the multi-head self-attention mech-
the autoregressive generation process because each token is pre-
anism. After 𝑀 self-attention blocks and a linear layer, the out-
dicted based on previously predicted tokens.
put sequences are represented by categorical vectors (denoted as
Unlike the RNN-related models where the current token are
𝒉) with 𝑁" dimensions, where each vector represents a probabil-
highly depends on previous tokens, the Transformer network
ity distribution over the activities in the vocabulary list.
uses the positional encoding technique to inform the relative
We applied argmax on 𝒉 to obtain one-hot representation of
positions of the tokens [26]. For this reason, the generation is
output sequences, where: 𝒔𝒊 = [𝒐𝟏 , 𝒐𝟐 , 𝒐𝟑 , . . . , 𝒐𝒍 ], 𝒐𝒊 =
more robust to the initial inputs and can generate all tokens of a
sequence in parallel. This non-autoregressive step alleviates the 𝑜𝑛𝑒ℎ𝑜𝑡(arg max 𝒉𝒊 )).
accumulated discrepancies when tokens are generated one by Discriminator. For discriminator 𝐷, we also use the Trans-
one and can handle longer, more complex sequences. In addition, former encoder to avoid mode collapse that may cause by the
with adversarial training, feeding Transformer with random se- learning rate mismatch between generator and discriminator
quences as input can regularize the entire model and reduce ex- during adversarial learning. We find the first end token for the
posure bias. generated sequence and pad all the tokens after it. 𝐷 takes se-
Data Preparation. For activities {𝑎! } in the process dataset, we quences (generated and authentic) of one-hot vectors as input
build an activity vocabulary 𝑉 (vocabulary size as 𝑁" ) using the and outputs a binary classification result.
Learning Objectives. The generator and discriminator were
index-based encoding. For each authentic process sequence 𝑥 =
optimized alternatively to achieve an equilibrium. The discrimi-
[𝑎# , 𝑎$ , … 𝑎% ], we pad a predefined end token [𝑒𝑛𝑑] to the end of
nator was used to score the generated sequences to train the
the sequence when the length is shorter than the maximal se-
generator. The generator’s objective is to maximize this score.
quence length 𝑙. Random input sequence 𝑧 = [𝑎# , 𝑎$ , … 𝑎& ] is
The output of the last layer of the generator is the sequences
generated by random sampling of the activity vocabulary (in-
of vectors 𝒉, with each dimension of 𝒉 representing the proba-
cluding the end token) until the max sequence length is reached.
bility of that token to be generated in the position. In the for-
ward propagation, we applied argmax on 𝒉 to generate a se- Evaluation of sequential data is not as intuitive as image data.
quence in one-hot encoding representation, which is then fed Synthetic images are usually evaluated based on the authenticity
into the discriminator. The argmax operation is not differentia- of characters or the resolution, which is easy to observe [6, 12].
ble in the back propagation, and the gradient cannot be passed We aim to generate synthetic process data that follows the un-
back. We adopted the straight-through Gumbel-Softmax mecha- derlying distribution of the real-world process where authentic
nism with a differentiable sampling operation [11]. The discrim- data is observed. Our evaluation takes a sample of synthetic se-
inator scores the generated sequences and passes the gradient quences and evaluates using three measures, i.e., statistical
back to the generator. The loss function of generator is: measures, supervised learning score, and workflow discovery.
Statistical Measures. We use three measurements to measure
ℒ/ = − 𝔼0~2(0) log 𝐷(G(Z)) (1)
the distance between synthetic and the authentic sequences.
where: 𝐺(𝑍) = 𝑆 = [𝒔𝟏 , 𝒔𝟐 , 𝒔𝟑 ,…, 𝒔𝒎 ]. (a) Sequence length: the number of activities in a sequence. We
We encoded the authentic sequences 𝑋 into one-hot repre- compared the mean and standard deviation of sampled syn-
sentation to make them the same expression as the generated thetic and authentic sequences.
sequences. The discriminator distinguishes between the generat- (b) Activity type occurrence: the count of occurrence of each
ed and the authentic sequences. The loss function of discrimina- activity type. Because each process data has different num-
tor is: ber of activity types, we reported an aggregated metric to
measure the difference of activity occurrence:
ℒ6 = − 𝔼7~2#$%$ log 𝐷(𝑋) <'

− 𝔼0~2(0) log(1 − 𝐷(𝐺(𝑍))) (2) 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝑋8 , 𝑆8 ) = \ |𝑋8 (𝑎! ) − 𝑆8 (𝑎! )|

!=#
Compared with RNNs, the random inputs and adversarial
where 𝑋8 , 𝑆8 represent the activity frequency distribution
training in ProcessGAN help reduce exposure bias. Due to the
of the datasets. The metric measures the L1 distance be-
larger search space, it is harder for the ProcessGAN to converge
tween the percentages of each activity type. We used L1
[8]. To help reduce the search space, we added activity distribu-
distance here for better interpretability instead of KL diver-
tion divergence as an auxiliary loss to the generator. The activity
gence.
distribution divergence was defined in two different ways, Kull-
(c) Sequence variance: the variance of sequences measures the
back-Leibler (KL) divergence and mean squared error (MSE):
spread or distance between sequences in a dataset. We used
# < 7’ (8 ) the sum of pairwise normalized edit distance (SPE) to meas-
ℒ/8 = 𝐾𝐿9!" (𝑋′8 ||𝑆′8 ) = ( ∑!=#
'
𝑆′8 (𝑎! ) log( ;’ $(8 &) ) (3)
$ &
ure variance:
<' # < <
ℒ/8 = 𝑀𝑆𝐸(𝑋′8 , 𝑆′8 ) = (×< ∑!=# [𝑋′8 (𝑎! ) − 𝑆′8 (𝑎! )]$ (4) 1
'
SPE = \ \ 𝐸𝐷(𝑠! , 𝑠B )/(length(𝑠! ) + length(𝑠B ))
where 𝑚 is the batch size, 𝑋′8 and 𝑆′8 are the activity probabil- 𝑁$
!=# B=!C#
ity distributions of one batch of authentic sequences and gener- where 𝑁 is the number of sequences in the generated sam-
ated sequences. We calculated the divergence in each training ples, 𝑠 are the sequences and 𝐸𝐷 is the edit distance (Le-
batch and feed it to generator (Figure 2). With divergence loss venshtein distance).
(ℒ/8 ), the generator loss becomes: These three measurements describe the basic dimensions of a
process sequence. The activity type occurrence is a cross meas-
ℒ/ ? = ℒ/ + 𝑤8 ℒ/8 (5)
urement, i.e., computed across synthetic and authentic process
where 𝑤8 is the weight of the auxiliary losses. sequences. The smaller distance indicates higher affinity be-
𝑤8 is a hyperparameter that balances adversarial loss and di- tween synthetic data and authentic data.
vergence loss. A heuristic choice of 𝑤8 is a value that enforces Supervised Learning Score. We can invite domain experts to
the two losses of similar scale. We first run the generator several manually evaluate the generated sequences. This manual as-
times without optimization and obtain the expectations of adver- sessment is labor-intensive and suffers from human bias. For this
sarial loss 𝔼(ℒ/ ) and divergence loss 𝔼(ℒ/8 ). 𝑤8 can be approx- reason, we trained an independent binary classifier and used it
𝔼(ℒ( ) as an “off-the-shelf” supervised scorer to assess the realism of
imated to $ .
𝔼(ℒ( )
the synthetic data. The classifier can be an evaluator with com-
Adversarial Training. At the early adversarial training stage, prehensive domain knowledge that can distinguish very similar
the 𝐷 may converge quickly because 𝐺 is generating implausible positive and negative sequences. We adopted the Transformer
sequences. 𝐷 may then reject all the generated sequences and model as the binary classifier. Traditional sequential classifica-
lead to poor optimization of 𝐺. To control the optimization speed tion models like RNN, CNN, or tree models with crafted features
and slow down 𝐷 ’s convergence, we adopted the training as input, can also be used as the classifier.
scheme of optimizing 𝑘 epochs (𝑘 set to 2 in our experiments) of Similar to the negative sampling method in the NLP field, we
𝐺 and one epoch of 𝐷 [6]. created high-quality activity sequences on the negative samples
by manually infusing noise (randomly adding, deleting, and
3.2 Evaluation Methods switching a predefined ratio of tokens) to the authentic sequenc-
es [15]. To distinguish more features of the data, after the Trans-
former layer we concatenated each sequence’s output with its 4 Experiment and Results
activity frequency distribution and sequence length and sent
them to two additional dense layers (Figure 3). Let 𝜃 be the pa- 4.1 Dataset Introduction
rameters of the model, the objective is to maximize the log- We performed our experiment and analysis on four medical
likelihood w.r.t 𝜃: process datasets of pediatric trauma resuscitation (the secondary
survey process, intubation process, emergency department pro-
𝜃 → \ log 𝑝(𝑦|𝑡, 𝜃)
cedures, and maintaining airway process). The process traces
D∈𝕋
were manually coded from videos. The use of these datasets was
where 𝕋 is the training set that contains the positive and nega- approved by the Institutional Review Board of Children’s Na-
tive samples, and 𝑦 is the correct class of 𝑡. tional Hospital in Washington, DC. In addition, we used two
We generated negative samples five times more than the pos- public process datasets2 to evaluate our method on different
itive samples to augment the training data. After the Transform- types of processes: a real-world event log which contains the
er achieved a good F1 (F1>0.8), we used it to score the synthetic process of sepsis cases from a hospital, and an event log of a loan
sequences. application process (Table 1). The loan application process has a
Process Mining. We applied the workflow discovery method large length variance. To test our model for generating process
from the process mining field to construct a workflow diagram with different degrees of complexity, we split this dataset into
from the generated process traces [28]. We aimed to visualize two parts: one with sequence lengths < 50 and the other > 50.
the process traces and evaluate whether they represented the
underlying distribution of authentic data. 4.2 Experimental Design
Baseline Methods. We used three traditional generative net-
“off-the-shelf” binary classifier
works for creating sequential data as baselines for evaluating of
Classification our ProcessGAN network: GRU, LSTM, and autoregressive
Transformer (Trans-AR). These networks sequentially generated
Sigmoid synthetic process traces.

Linear
Table 1: Statistics of four medical datasets and two public
datasets, including the case number, mean value and
Concat standard deviation of the sequence length, and the number
of activity types. (D1: Secondary survey process of trauma
resuscitation; D2: Intubation process of trauma resuscita-
Linear Linear
tion; D3: Emergency department process; D4: Maintain
Concat
airway process of trauma resuscitation; D5: Sepsis process;
Transformer Encoder D6: Loan application process dataset, where the first part
contains the sequences with length < 50 and the second
Activity Distribution Length Input Sequence part contains the sequences with length > 50.)
D1 D2 D3 D4 D5 D6_1 D6_2
# Cases 271 101 381 53 846 3271 1013
Authentic
Length (Mean) 46.03 12.35 5.53 31.72 17.28 34.50 68.89
Negative
Sequences Sequences Length (Std.) 15.20 3.18 2.13 18.08 12.01 8.58 18.03
# Act. Types 47 15 55 38 17 37 37
1 2 3 4 5 … 2 4 3 1 5 …

Deleted Switched Added

Our Methods. We first tried a non-autoregressive Transformer
to generate sequences using random sequences as input without
Figure 3: To generate negative sequences, we ran- GAN structure. We then used the proposed ProcessGAN model
domly add noises to the authentic sequence. The in- and two of its variants to generate synthetic process traces.
put of the classifier is trained based on the sequences, • Trans-NAR: A non-autoregressive Transformer network that
their activity distribution and length.
uses random sequences with fixed length as input and gener-
To construct the diagram, we used the trace alignment meth- ates synthetic process traces in parallel.
od to generate consensus sequence from the alignment result [4]. • P-GAN: The vanilla ProcessGAN model with adversarial
A consensus sequence captures the major activities in the pro- training only. It is a GAN-based Transformer network that
cess traces and can be considered the workflow diagram’s back- uses random sequences with fixed length as input and gener-
bone. Some activities that are not in the consensus sequence but ates synthetic process traces in parallel.
are required are added as the side branches in addition to the • P-GAN-M: P-GAN variant with the mean squared error loss
backbone to represent the parallel activities, and some infre- as activity distribution loss.
quent activities are filtered out [13]. We asked domain experts to
compare the workflow diagrams and check for process errors. 2 [Link]
 Table 2: The sequence lengths of the process data generated by different models.
D1 D2 D3 D4 D5 D6_1 D6_2
Authentic 46.89±12.46 13.3±2.69 5.68±2.07 31.0±23.49 17.53±9.50 34.85±8.71 67.87±17.45
GRU 42.33±22.15 11.37±3.46 5.20±2.14 28.27±20.43 15.78±7.42 32.88±9.60 70.87±34.05
LSTM 44.68±21.02 11.41±2.87 5.21±2.11 20.18±16.47 15.82±8.36 32.18±9.10 68.17±33.76
Trans-AR 39.38±18.27 11.36±2.86 5.08±1.92 25.89±14.06 16.71±7.59 33.25±8.81 56.56±26.80
Trans-NAR 31.53±5.38 11.27±1.64 4.56±1.41 12.90±4.14 11.97±2.68 24.86±4.86 56.68±6.05
P-GAN 43.79±13.32 12.40±1.76 5.31±1.92 33.09±15.91 17.19±10.37 33.78±7.98 63.29±15.57
P-GAN-M 43.49±11.79 12.70±1.69 5.74±1.76 35.82±9.54 17.25±8.47 34.84±7.82 68.62±19.87
P-GAN-K 46.24±18.13 12.21±2.02 5.58±1.64 29.71±22.57 17.10±5.18 33.59±9.48 64.59±6.61

• P-GAN-K: P-GAN variant with the KL divergence loss as ac- quences with smaller standard deviations on all the datasets.
tivity distribution loss. Non-autoregressive Transformer model was trained by minimiz-
The baseline models are all autoregressive models. These ing the cross-entropy loss between the full generated sequences
models were trained through the authentic sequences by using and the authentic sequences directly. It generated tokens with
the ground truth tokens to predict the next token in a sequence, the assumption of independence, which is not reasonable for se-
i.e., ∏I!=# 𝑃(𝑎!C# |𝑎! , 𝑎!G# , … , 𝑎# , 𝑎H ). For the Trans-NAR model, quential data generation [7]. The model usually converges faster
we stopped training when the loss converged. It can be difficult than other models but fails to generate precise sequences. The
for GAN models to tell the stop point, especially for sequential GAN-based model that incorporates an additional discriminator
data that are not as intuitive as images. The loss value of both 𝐺 can help the generator find sequences with more precise activity
and 𝐷 were oscillating because they were playing a minimax orders, which addresses the independence assumption problem.
game. Ideally, the accuracy of 𝐷 will first rise because 𝐺 is poor.
As the performance of 𝐺 gets better, 𝐷’s accuracy will drop. If Table 4: The sequence variance (SPE) of authentic data
the accuracy is oscillating around 0.5, 𝐺 is generating plausible and synthetic data generated by different models. Syn-
thetic sequence variance that is closer to authentic data
sequences to tricks 𝐷. For this reason, we picked the result when
is considered to be better.
𝐷 and 𝐺 reach this equilibrium [6].
We divided each dataset into training, validation, and testing D1 D2 D3 D4 D5 D6_1 D6_2
Authentic 0.21 0.14 0.21 0.23 0.17 0.16 0.17
sets in a 0.8:0.1:0.1 ratio. For each model, we generated 500 syn-
GRU 0.28 0.19 0.24 0.31 0.17 0.17 0.21
thetic sequences and compared them with the hold-out test set. LSTM 0.27 0.17 0.24 0.32 0.18 0.17 0.21
Trans-AR 0.29 0.20 0.25 0.29 0.17 0.18 0.23
Trans-NAR 0.24 0.19 0.24 0.25 0.16 0.19 0.20
Table 3: The overall activity type occurrence differ- P-GAN 0.24 0.18 0.23 0.26 0.17 0.19 0.18
ences of the sequences generated by different models. P-GAN-M 0.25 0.18 0.22 0.23 0.17 0.15 0.19
P-GAN-K 0.26 0.19 0.22 0.26 0.15 0.20 0.18
D1 D2 D3 D4 D5 D6_1 D6_2
GRU 0.11 0.16 0.23 0.64 0.09 0.10 0.08 We found that the autoregressive models perform better for
LSTM 0.11 0.17 0.24 0.68 0.08 0.14 0.06 activity type occurrence (Table 3). This observation is expected
Trans-AR 0.19 0.17 0.20 0.58 0.11 0.13 0.10
Trans-NAR 0.24 0.18 0.31 0.55 0.32 0.37 0.17
because the models are trained with the MLE objective through
P-GAN 0.31 0.23 0.45 0.77 0.15 0.34 0.32 teacher forcing, in which case the activity distribution of the
P-GAN-M 0.17 0.21 0.29 0.55 0.07 0.22 0.17 training data can be easily learned. Vanilla ProcessGAN shows a
P-GAN-K 0.11 0.16 0.23 0.60 0.11 0.11 0.28
large difference in activity type occurrence with authentic data.
With activity distribution divergence as an auxiliary loss, P-
4.3 Results GAN-M and P-GAN-K also generated comparable activity distri-
4.3.1 Quantitative results
Table 5: The fraction of the synthetic sequences that the
Statistics. We repeated each experiment several times and “off-the-shelf” binary classifier classified as authentic. This
found the results are stable. For the sequence lengths (Table 2), fraction measures how frequently the synthetic data
we found the ProcessGAN-based models outperformed other “tricks” the binary classifier. And it is formally defined as
models without a GAN structure, especially for the datasets with false positive rate (FPR = FP/N), where FP is the count of
longer sequences. The discriminator of the GAN networks can synthetic sequences misclassified as authentic, and N is the
help the generator avoid generating biased sequences that are count of all negatives, i.e., synthetic sequences.
either too long or too short. For longer sequences with smaller D1 D2 D3 D4 D5 D6_1 D6_2
data sizes (D1, D4, D6_2), we observed that the RNN-based mod- F1 0.94 0.95 0.93 0.89 0.98 0.96 0.98
els and the autoregressive Transformer model tend to generate GRU 75.6% 82.4% 91.2% 63.4% 94.8% 91.6% 87.6%
LSTM 78.4% 82.4% 89.8% 47.4% 91.0% 90.4% 86.8%
sequences with shorter lengths or a larger standard deviation Trans-AR 72.2% 64.4% 90.0% 66.2% 80.0% 88.4% 82.4%
(Figure 4). For shorter sequences (D2, D3) or sequences with rel- Trans-NAR 65.8% 65.8% 84.6% 21.6% 81.8% 78.8% 72.6%
atively sufficient data size (D5, D6_1), the autoregressive models P-GAN 80.0% 90.4% 96.2% 71.6% 75.8% 84.0% 97.6%
P-GAN-M 84.4% 77.2% 97.2% 72.6% 95.0% 75.6% 98.0%
can generate comparable sequence lengths. We also observed the P-GAN-K 70.4% 89.4% 98.2% 87.2% 97.2% 77.0% 94.4%
non-autoregressive Transformer model generated shorter se-
 GRU LSTM Trans-NAR

P-GAN P-GAN-M P-GAN-K

Figure 4: The length distribution of the synthetic sequences generated by different models for dataset D1. The x-axis
represents the sequence length (i.e., the number of activities), and the y-axis represents the fraction of each length in
the dataset.

GRU LSTM Trans-NAR

Authentic Data
(1)

P-GAN P-GAN-M P-GAN-K

(2)

Figure 5: (1) The activity occurrence distributions of the authentic sequences in dataset D1. (2) The difference of activity
occurrence frequency between authentic and synthetic sequences generated by different models. The x-axis represents
the activity types, and the y-axis represents the difference.

bution to the autoregressive models (Table 3, Figure 5). For com- Supervised Learning Score. We used the false positive rate
plex processes (i.e., longer sequence and large activity vocabu- (percentage of the synthetic sequences being classified as au-
lary) with very limited data size (D4), the autoregressive models thentic) as the metric to measure the authenticity (Table 5). The
did not have an advantage in generating a good activity distribu- more sequences being classified as positive, the better our pro-
tion. The performance of these models relied on sufficient data. cess generator performs.
From the sequence variance perspective (Table 4), we found We found the GAN-based models have better scores than
that all models can generate variable sequences. When the data other models for most of the datasets, which means the activities
size is large (D5, D6_1), all models performed about the same. in these sequences have a more reasonable order. Considering
When the amount of process data was small or more complex, that the Transformer-based classification model can capture
the GAN based models performed better in generating sequences more intermediate dependencies between the tokens, a higher
with similar variance to the authentic data. score proved that our models managed generating sequences
From the statistical results, we found that the process data with complex activity dependencies. We found the models with
complexity and the data size can impact the performance of dif- better results in activity type occurrence and sequence variance
ferent models. We found the GAN-based models provide more did not always get higher scores. The models capturing a good
regularizations when sequences are complex and when data size global similarity between the sequences does not mean they cap-
is insufficient, which makes these models suitable for process tured good local activity dependencies. We found that in D6_1,
data augmentation in such conditions. Autoregressive generative the GAN-based models got lower score and the baseline auto-
models are better at mimicking the observed process data. regressive models achieved higher scores, and in D6_2 when we
increased the process length, the score of the baseline RNN-
 START(1.0)
START(1.0)

Patient Arrival(0.93)
Patient Arrival(0.89) BVM(0.89) Decision to Intubate(0.76) NRB(0.89) Pre-Oxygenation Breathing Verbalized(0.62) Pre-Oxygenation Chest Auscultation(0.71)

NRB(0.35)
Critical Window(0.7) RSI Sedative Meds_2(0.61)

START(1.0)
RSI Sedative Meds(0.66)
BVM_2(0.61) Pre-Oxygenation Chest Auscultation(0.57) Airway Assessment(0.62)
START(1.0)

Patient Arrival(0.89)
RSI Paralytic Meds(0.79) Decision to Intubate(0.47) Pre-Oxygenation Breathing Verbalized(0.62)
Patient Arrival(0.93)

Airway Assessment(0.44)
Laryngoscopy(0.72) BVM(0.39) NRB_2(0.67)
BVM(0.42)

Pre-Oxygenation Chest Auscultation(0.6)

END(1.0) Critical Window(0.66)
Pre-Oxygenation Chest Auscultation(0.76) NRB(0.86) Pre-Oxygenation Breathing Verbalized(0.65)

GRU Workflow BVM_2(0.75) Decision to Intubate(0.41) Pre-Oxygenation Breathing Verbalized(0.62)

Decision to Intubate(0.48) BVM_2(0.72) RSI Sedative Meds(0.65)

Critical Window(0.61) NRB(0.61)

RSI Paralytic Meds(0.65)

RSI Sedative Meds(0.93) BVM(0.38)

Laryngoscopy(0.56)

RSI Paralytic Meds(0.78)

NRB_2(0.68) Critical Window(0.74) Decision to Intubate_2(0.66)
END(1.0)

Laryngoscopy(0.71)
RSI Sedative Meds(0.43)
GRU + ProcessGAN Workflow
END(1.0)
Laryngoscopy_2(0.61) RSI Paralytic Meds(0.5)

Authentic Workflow
Critical Window_2(0.62) Laryngoscopy(0.54)

END(1.0)

ProcessGAN Workflow

Figure 6: The workflow diagrams of the authentic Intubation dataset (D2) and three synthetic ones discovered by the
alignment method [4]. The gray ovals are the backbone activities that represent the major process, and the white ovals
are the side-branch activities that represent the parallel activities. The numbers in the brackets represent the frequency
of the activity. The mismatches are marked with red signs (tick: potential correct order, cross: incorrect order).

related models dropped and were surpassed by the GAN-based 3. An additional “RSI Sedative Meds” activity spanned be-
models, which means the increase of process complexity reduces tween the start point and the “RSI Paralytic Meds” activity
the performance of RNN-related models. Our GAN-based models in the GRU workflow. Our experts state that “RSI Sedative
are more robust with different process complexity. The Trans- Meds” must immediately precede “RSI Paralytic Meds” and
former model also outperformed the RNN-related models be- cannot span from the start point and “RSI Paralytic Meds”.
cause the multi-head attention mechanism achieves better per- 4. Additional “Laryngoscopy” and “Critical Window” activi-
formance in longer sequences. ties spanned the entire diagram in the ProcessGAN work-
flow. These activities cannot span the entire process based
4.3.2 Case Study
on domain knowledge.
Workflow Discovery. We used the Intubation dataset (D2) as a The medical experts found that both synthetic workflows
case study to discover the workflow diagrams and further evalu- captured the major treatment steps of intubation. As RNN is not
ate the quality of the synthetic process data. We generated work- good at capturing long-range dependencies, GRU tended to gen-
flows for the authentic traces, the GRU-generated traces and the erate shorter traces (Figure 4, upper left corner). GRU workflow
ProcessGAN-generated traces. We also combined the synthetic performed well by representing the activities in correct positions
traces of both models (Figure 6). In synthetic workflows, the or- of the process. From the GRU workflow (Figure 6), we observed
der of “Patient Arrival”, “BVM”, “Decision to Intubate”, “Critical that all the activities before “Critical Window” could be executed
Window”, “RSI Sedative Meds”, “RSI Paralytic Meds” and “Lar- in parallel, and the activities after it require a logical ordering.
yngoscopy” activities matched the authentic workflow. The GRU traces fit the real clinical scenarios, although some
We observed some discrepancies between the synthetic and traces have “RSI Sedative Meds” in an erroneous position.
authentic workflows, including different activity orders and po- The ProcessGAN workflow performed well in giving more
sitions. To address these, we invited medical experts to evaluate detailed workflow. The medical experts liked the ProcessGAN
the clinical applicability of the GRU and ProcessGAN synthetic workflow because it included longer traces and a variety of plau-
workflows: sible pathways. Although the ProcessGAN generated more
1. The activities “NRB'' and “Pre-Oxygenation Breathing Ver- pathways that were different from the authentic traces, these
balized” precede “Decision to Intubate” in the authentic pathways are valid according to expert knowledge. ProcessGAN
workflow. We observed these activities followed the “Deci- expanded the probability of occurrence of valid pathways. It is
sion to Intubate” in the ProcessGAN workflow. Our medi- not learning to “copy” the authentic data but learning to “imi-
cal experts confirmed these activities could happen follow- tate” the structural properties of the authentic data. It can gener-
ing the decision to intubate. ate more unobserved traces and augment the process datasets.
2. The “Airway Assessment” activity appears in the Pro- According to the ProcessGAN workflow, “Laryngoscopy” and
cessGAN workflow and not in the authentic workflow. Ac- “Critical Window” activities dispersed out of their consensus
cording to the medical experts, airway assessment is in the sequence position (Figure 6) each in 62% and 61% of the synthet-
correct position and routinely performed [18]. ic traces. Similarly, “RSI Sedative Meds” activity occurs in the
GRU workflow in 61% of synthetic traces dispersed out of its [9] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu and Jun Wang. 2018.
Long text generation via adversarial training with leaked information. In
consensus sequence position. These occurrences represent pro- Proceedings of the AAAI Conference on Artificial Intelligence.
cess errors, so these traces are invalid. The invalid traces can be [10] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.
Neural Computation, Vol. 9. 8 (1997), 1735-1780.
rejected or rectified using rules derived from domain knowledge [11] Eric Jang, Shixiang Gu and Ben Poole. 2016. Categorical reparameterization
[18]. Note that in a general case even the authentic dataset may with gumbel-softmax. arXiv preprint arXiv:1611.01144, (2016).
contain process errors. [12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen and
Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In
When the traces generated by both GRU and ProcessGAN Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
models are combined, the discovered process workflow for the Recognition. 8110-8119.
[13] Jingyuan Li, Sen Yang, Shuhong Chen, Fei Tao, Ivan Marsic and Randall S
combined dataset was the most similar to the authentic process Burd. 2018. Discovering interpretable medical workflow models. In 2018 IEEE
workflow (Figure 6). The pathways of the combined dataset International Conference on Healthcare Informatics (ICHI). 437-439.
workflow are all semantically valid, improving the authenticity [14] Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter and Dan
Jurafsky. 2017. Adversarial Learning for Neural Dialogue Generation. arXiv preprint
of the synthetic dataset. Our future work will investigate the arXiv:1701.06547, (2017).
methods for training different generative models together to im- [15] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado and Jeff Dean. 2013.
Distributed representations of words and phrases and their compositionality. In
prove the quality of synthetic datasets. Advances in Neural Information Processing Systems. 3111-3119.
[16] Dominic A Neu, Johannes Lahann and Peter Fettke. 2021. A systematic
literature review on state-of-the-art deep learning methods for process prediction.
5 Conclusion Artificial Intelligence Review, (2021), 1-27.
[17] Weili Nie, Nina Narodytska and Ankit Patel. 2018. Relgan: Relational
Due to the privacy and confidentiality concerns, most process generative adversarial networks for text generation. In International conference on
data cannot be shared publicly. This work tries to resolve this learning representations.
problem by proposing a framework for generating synthetic pro- [18] Karen J O’Connell, Sen Yang, Megan Cheng, Alexis B Sandler, Niall H
Cochrane, JaeWon Yang, Rachel B Webman, Ivan Marsic and Randall Burd. 2019.
cess data. The framework works by first learning deep represen- Process conformance is associated with successful first intubation attempt and
tation of process data and uses the learnt model as a generative lower odds of adverse events in a paediatric emergency setting. Emergency Medicine
Journal, Vol. 36. 9 (2019), 520-528.
model for generating synthetic process data that can be shared [19] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli and Wojciech Zaremba.
publicly. We tested popular deep sequential models, i.e., GRU, 2015. Sequence level training with recurrent neural networks. arXiv preprint
LSTM, and Transformer. We also introduced ProcessGAN, a arXiv:1511.06732, (2015).
[20] Eric Rojas, Jorge Munoz-Gama, Marcos Sepúlveda and Daniel Capurro. 2016.
generative adversarial network of two non-autoregressive Process mining in healthcare: A literature review. Journal of Biomedical Informatics,
Transformers with customized loss function for process data. In Vol. 61 (2016), 224-236.
[21] Alex Sherstinsky. 2020. Fundamentals of recurrent neural network (RNN) and
addition, comprehensive evaluation methods were proposed to long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, Vol.
assess the quality of the generated process data. Our experiments 404 (2020), 132306.
[22] Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data
show that, compared to other models, ProcessGAN can augment augmentation for deep learning. Journal of Big Data, Vol. 6. 1 (2019), 1-48.
the existing process dataset and perform better in the cases of [23] Latanya Sweeney. 2000. Simple demographics often identify people uniquely.
complex process data with small accessible data. Traditional Health (San Francisco), Vol. 671. 2000 (2000), 1-34.
[24] Farbod Taymouri, Marcello La Rosa, Sarah Erfani, Zahra Dasht Bozorgi and
deep sequential generative models on the other hand can gener- Ilya Verenich. 2020. Predictive business process monitoring via generative
ate process sequences that are more like observations. Overall, adversarial nets: The case of next event prediction. In International Conference on
Business Process Management. 237-256.
the augmented process data largely reduce the effort of data col- [25] Wil MP Van Der Aalst, Hajo A Reijers, Anton JMM Weijters, Boudewijn F van
lection and can be shared publicly. Dongen, AK Alves De Medeiros, Minseok Song and HMW Verbeek. 2007. Business
process mining: An industrial application. Information Systems, Vol. 32. 5 (2007),
713-732.
REFERENCES [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
[1] Samy Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer. 2015. Scheduled Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need.
sampling for sequence prediction with recurrent neural networks. arXiv preprint In Advances in Neural Information Processing Systems. 5998-6008.
arXiv:1506.03099, (2015). [27] Zhenchen Wang, Puja Myles and Allan Tucker. 2019. Generating and
[2] Zaharah A Bukhsh, Aaqib Saeed and Remco M Dijkman. 2021. evaluating synthetic UK primary care data: preserving data utility & patient
ProcessTransformer: Predictive Business Process Monitoring with Transformer privacy. In 2019 IEEE 32nd International Symposium on Computer-Based Medical
Network. arXiv preprint arXiv:2104.00721, (2021). Systems (CBMS). 126-131.
[3] Awatef HICHEUR Cairns, Billel Gueni, Mehdi Fhima, Andrew Cairns, Stéphane [28] AJMM Weijters and Wil MP van der Aalst. 2001. Process mining: discovering
David and Nasser Khelifa. 2015. Process mining in the education domain. workflow models from event-based data. In Belgium-Netherlands Conf. on
International Journal on Advances in Intelligent Systems, Vol. 8. 1 (2015), 219-232. Artificial Intelligence.
[4] Shuhong Chen, Sen Yang, Moliang Zhou, Randall Burd and Ivan Marsic. 2017. [29] Sen Yang, Yichen Zhou, Yifeng Guo, Richard A Farneth, Ivan Marsic and Burd
Process-oriented iterative multiple alignment for medical process mining. In 2017 S Randall. 2017. Semi-synthetic trauma resuscitation process data generator. In 2017
IEEE international conference on data mining workshops (ICDMW). 438-445. IEEE International Conference on Healthcare Informatics (ICHI). 573-573.
[5] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau and Yoshua Bengio. [30] Lantao Yu, Weinan Zhang, Jun Wang and Yong Yu. 2017. Seqgan: Sequence
2014. On the properties of neural machine translation: Encoder-decoder generative adversarial nets with policy gradient. In Proceedings of the AAAI
approaches. arXiv preprint arXiv:1409.1259, (2014). conference on artificial intelligence.
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- [31] Yi Yu, Abhishek Srivastava and Simon Canales. 2021. Conditional lstm-gan for
Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio. 2014. Generative melody generation from lyrics. ACM Transactions on Multimedia Computing,
adversarial nets. Advances in Neural Information Processing Systems, Vol. 27 (2014). Communications, and Applications (TOMM), Vol. 17. 1 (2021), 1-20.
[7] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li and Richard Socher. [32] Yizhe Zhang, Zhe Gan and Lawrence Carin. 2016. Generating text via
2017. Non-autoregressive neural machine translation. arXiv preprint adversarial training. In NIPS workshop on Adversarial Training. 21-32.
arXiv:1711.02281, (2017).
[8] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin and Aaron
Courville. 2017. Improved training of wasserstein gans. arXiv preprint
arXiv:1704.00028, (2017).