Objective
During research, we faced real problems when trying to use decision-data mining models in
BI systems. We saw it was mostly clear in retail chains, where very huge datasets like the
Corporación Favorita dataset put overload on traditional systems because of millions of
records. These systems often face problems with identifying hidden relationships between
variables and also data decay.
We found that existing framework like Elovici and Braha (2003) provide useful economic
evaluations through use of net loss bounds. But these are difficult to apply in practical
situations without human intervention. We propose an integrated and automated approach to
improve both analytical accuracy and economic efficiency in real-time Business Intelligence
(BI) pipelines. Our motivation comes from practical cases in retail and supply chain
environments, where unnoticed data quality drift can lead to poor inventory decisions, higher
operational costs, and reduced decision outcomes.
To demonstrate its practical value, we define four main objectives that address the limitations
of existing approaches and support scalable deployment in Business Intelligence (BI)
systems:
Our first objective is automating multi-dimensional correlation discovery without manual
feature engineering. We aim to use variational encoder to identify complex relationships
between multiple variables-such as sales volume, transaction spikes, oil price fluctuations,
and holiday effects-and represent them in lower dimension latent space. The orthodox
approaches that rely on manual transformations (e.g., NFS from Chen et al., 2019), our
method works directly on primary sequences. Our goal was to get over 95% correlation
accuracy, measured using normalized mutual information, while reducing preprocessing time
from days to hours in our experiments.
Probabilistic Forecasting of Data Quality (DQ) Degradation is another objective. We noticed
lack of temporal prediction in earlier utility-based models (Crone at al., 2005), and integrated
bidirectional LSTM with VAE decoding to forecast future changes in data quality indicators
such as completeness (e.g., missing oil price entries) and consistency (e.g., changes in
promotion flags.). The target performance was MSE below 0.025 for 7-day prediction
horizons with 90% confidence intervals. This allows early alerts that could prevent an
estimated 15-20% of decision payoff losses in artificial drift framework, based on Monte
Carlo experiments.
Our third objective is to carefully evaluate the framework on the Corporación Favorita
dataset. We estimate reconstruction quality through per-window mean squared error, anomaly
detection through 95th-percentile threshold (practically determined as 0.010004 on the test
set). DQ degradation predicting through a 3 step ahead completeness alert system. We report
all results clearly. This includes any limitations seen in the inference outputs. We discuss
what these findings mean for model calibration also and threshold design when applied in
real-world situations.
Our fourth goal should be to systematically remove different parts of the model to understand
their contribution. These studies will isolate the result of each part of the architecture: the
variational encoder, the KL divergence regularizer at B = 0.005, the z-conditioned future
forecasting loop, and the combined three-term loss. This approach helps us, it connects
observed performance to specific design choices, rather than the overall capacity of the
model.
Organization of the Paper
The paper we wrote, follows natural path from motivation through methodology to evaluation
and each section is crafted directly upon the one preceding it.
The Literature Review section highlights the explanatory survey of twenty works in decision
theoretic data mining, online learning, ongoing analytics, preprocessing automation and
model diagnostic. We found that its contribution to field and its limitations that leaves
unaddressed with respect to multi-dimensional correlation learning and time-related DQ
forecasting.
Experimental Infrastructure and Technical Specifications describes our rig—AWS GPUs,
PyTorch stacks, Ray Tune sweeps and provides a complete description of all hardware,
software, hyperparameters, and choices of configurations made. We also report the actual
training environment (CPU), the real loss values detected at each checkpoint, and the original
data volume of 1,660 windows produced from the Favorita dataset.
Dataset Description zooms into characterization of the Corporación Favorita dataset that is
3M+ sales ticks, oil gaps, holiday, and also making clear each merging and imputation
decision, including the linear interpolation used for weekend oil price gaps.
The Experimentation, Results, and Analysis section reports all experimental outcomes that
are including the real training loss trajectory, the reconstruction anomaly threshold of
0.010004, the five different windows identified in the test set (Windows 59, 60, 61, 62, and
68), the completeness forecasting behavior observed across 100 test windows, and the
limitations this shows about threshold calibration for DQ alerting.
The Conclusion and Future Scope section synthesises contributions and identifies four
directions for future work.
Literature Review
We started looking into decision-theoretic data mining. We were surprised by how nicely it
connects getting knowledge from data to actual financial gains. A system which
automatically finds correlations in multi-dimensional data and also predicts when the data
will start degrading further, is not something that drives us from a single idea. What we will
talk about next is not a list but rather a focused vision of the main ideas. Each of these ideas
has produced truly useful work, yet none of them, by taking alone, covers the whole problem.
Twenty papers are reviewed below, and categorized into two subject oriented groups. The
two papers most directly relevant to our idea - Malhotra et al. [19] on LSTM-based encoder-
decoder anomaly detection, and Luo et al. [20] on dual-stage attention LSTM-VAE for KPI
anomaly detection — are treated in their own dedicated subsection because they are the
closest technical predecessors to what we propose.
This group of research forms the theoretical foundation or we can say theoretical backbone
for evaluating predictions in data-driven systems. The main question these papers revolve
around is: when a system keeps making predictions over time and the decisions it take, how
can we measure if it is performing well and how should it update itself based on new
information?
Hedge algorithm was introduced by them (Freund and Schapire [1]), which is an important
method in online learning. The algorithm works by maintaining a probability distribution
over a collection of expert strategies and updating that distribution multiplicatively at each
step based on each expert's observed loss. The algorithm guarantees that its total error will be
close to the best expert in hindsight. There is a limitation, which matter for our purposes. The
algorithm requires the loss values and variables to provided manually by the user. User must
decide which variables the algorithm the system should consider.
In 1997, Freund and Schapire [2] extended their work, which was to build on the Hedge
framework to formalise boosting as a decision-theoretic process, demonstrating that weak
learners — classifiers performing only marginally better than random chance — could be
combined into an arbitrarily strong ensemble under the same regret-minimisation guarantee.
The idea led to AdaBoost, which became widely used in machine learning applications. The
limitation is still unchanged, boosting still assumes that the input features are already defined.
The correlation discovery problem — identifying which of many possible variable
interactions carry predictive signal for DQ degradation — is entirely outside the scope of this
work.
Elovici and Braha [6] proposed a most directly relevant framing for evaluating data mining
system in operational settings. Rather than measuring success through prediction accuracy
alone, they propose net loss mininisation relative to the best available strategy as the primary
criterion - a foumulation that correctly captures the asymmetric costs of acting on corrupted
versus clean datain high-stakes BI environments. The financial cost of making a wrong
decision can be higher than the prediction error itself, that's why this perspective is important
for anybody. That is why we take this very economic framing as part of our motivation for
proactive DQ forecasting. However, their framework do not address how to acquire or
validate it automatically, nor do they propose any mechanism to predict data quality problems
before they affect decisions.
Several other papers represent the mathematical maturation of the expert learning tradition
through the 1990s. Vvok [14] developed the aggregating algorithm and its associated loss
bounds, providing a unified framework that subsumes Hedge as a special case and extends to
a broader class of loss functions. Kivinen and Warmuth [15] introduced the exponentiated
gradient (EG) algorithm, which works well with high-dimensional sparse data. A result that
became influential in large-scale text and recommendation applications. Haussler, Kearns and
Schapire [16] analysed the cumulative loss of multiplicative weight algorithms using a game-
theoretic treatment to sequential prediction under partial information. Chung[17] studied
sequential prediction using game theory under partial information.
Kingma and Welling [13], they introduced Variational Autoencoder (VAE), laying the
theoretical and computational foundations. Very important method for learning finding
hidden representation of data without the help of supervision. Maximising the log-likelihood
log p(x) is intractable for deep generative models, core derivation showed this but it can
replaced by maximising the ELBO (Evidence Lower Bound): L θ, ϕ; x) = E[qϕ(z|x)][log
pθ(x|z)] − D_KL(qϕ(z|x) || p(z)), where qϕ(z|x) is the encoder's approximate following, pθ(x|
z) is the decoder's generative model, and p(z) = N(0, I) is the standard normal prior.
The VAE is crucial as it is the direct mathematical for the correlation-learning component of
our system. The VAE consist of two parts, input data is converted into latent representation
by Encoder and another one is Decoder which reconstructs the original data from that latent
vector. A major innovation in their work was the reparameterization trick, which let the
model to be trained efficiently using standard backpropagation.
What it does not provide is the original VAE treats input as an independent observation
drawn from a static distribution. If we applied a normal VAE directly to time-series data, we
would lose important information like the order of events, gradual changes over time and
early signals of data quality problems.
Malhotra, Ramakrishnan, Anand, Vig, Agarwal, and Shroff presented an LSTM-based
encoder-decoder architecture for detecting anomalies in multi-sensor data. The approach is
structurally intuitive: an LSTM encoder reads a sequence of sensor data and compresses it
into a hidden representation, a symmetric LSTM decoder then attempts to reconstruct the
original window from that vector. The model is trained on only on normal data.
Our architecture follows the same encoder-decoder structure. The pattern, processing a fixed-
length temporal window through an LSTM encoder, then reconstructing it through a
symmetric LSTM decoder - is the template our architecture follows. In our case the system
analyzes seven data-quality dimensions. Our reconstruction anomaly threshold of 0.010004
was set at the 95th percentile of reconstruction MSE across the 100-window test set.
The main limitation of Malhotra et al. [19] from our approach is that it is deterministic nature
of their encoder. The encoder produces a single fixed vector, not a probability distribution.
This has two problems. First, there is no probabilistic interpretation of anomaly score and
second and more critically it cannot predict future states of the system.
Luo, Zhang, Hu, and Zhang [20] published a dual-stage attention-based LSTM-VAE for Key
Performance Indicator (KPI) anomaly detection. Their model combines LSTM (to capture
time dependencies), VAE (to learn probabilistic latent representations) and Attention
mechanisms (to focus on important parts of the data). Their dual-stage attention mechanism
operates at two levels: a temporal attention stage that learns soft weights over important time
steps and a feature attention stage that learns and focuses on the most relevant input variables,
allowing model to down weight noisy or less relevant KPI channels during encoding.
The results reported by Luo et al. [20] are strong across standard anomaly detection
benchmarks. It benefits are better generalization due to the VAE latent space and improved
interpretability because attention shows which parts of the data were important. Both
contributions are genuine.
The limitation that separates Luo et al. [20] from our work is the same one that separates
virtually all anomaly detection literature from DQ forecasting. It identifies anomalies after
they have already occurred. The framework is reactive than predictive. It does not generate
predictions about what future windows will contain, nor does it issue warnings before
degradation is already observable. We believe, this delay can be costly because the corrupted
data may already have influenced business decision.
Dataset Description
Corporación Favorita Grocery Sales Forecasting dataset is a publicly available retail
benchmark freely accessible at [Link]
It covers the period from January 1,2013 to August 15,2017, across 33 product families and
54 stores. Five year span of 1,684 continuous daily observation gives sufficient temporal
depth for sequence modelling. This dataset is suitable for our study as it contains real multi-
dimensional relationships between macroeconomic, promotional, calendar, and operational
factors. It expands over five years, which is more than enough sequence length for temporal
modelling; and it includes naturally occurring data quality (DQ) issues - such as missing oil
prices on weekends, label noise caused by transferred holidays, and drift in promotion flags -
that a DQ forecasting system must learn to predict in advance rather than just respond to.
Six files were combined into one daily analytical dataset. The main file is [Link] (with
more than 3 million records), provides daily unit sales and promotion counts for all store and
product family combinations. The [Link] file has metadata about store types (A-E) and
clusters (1-17). The [Link] file contains daily Ecuadorian crude oil prices, where missing
weekend values were filled using linear interpolation. The holidays_events.csv file lists more
than 350 calendar events, including transferred holidays, which shift sales patterns to random
or unexpected dates and are source of label noise. The [Link] file provides daily
customer footfall data each store.
From this combined dataset, seven data quality (DQ) dimensions were created using only
group level aggregations. There were no manually designed interaction features or domain
expert feature combinations were used.
Id DQ Dimension Derivation Quality Property
x
0 dq_completeness_stores active_stores / Fraction of stores reporting on
54 that day
1 dq_completeness_families active_families Fraction of product families
/ 33 reporting
2 dq_volume Sum of daily Pipeline volume — feed
sales integrity proxy
3 dq_promotion_rate Mean Promotional data availability
onpromotion
4 dq_oil_price dcoilwtico Macroeconomic signal
(imputed) completeness
5 dq_transaction_count Sum of Transaction feed activity
transactions
6 dq_holiday_impact Daily event Calendar coverage for label
count correctness
B. Sequence Construction and Evaluations Split
The seven-column dataset was normalised to range [0,1] via MinMaxScaler. It was then
divided using a 24- day sliding window with a with stride one. resulting in a final tensor of
shape (1660, 24, 7). Each window represents a continuous 24-day sequence of DQ signal
(which is seven-dimensional). The last100 windows were kept and used as a test set, and first
1,560 windows were used for training.
Proposed Methodology
The LSTM-VAE framework solves two core limitations identified in the literature review
within a single combined architecture. A compact and probabilistic representation of multi-
dimensional DQ relationships without any manual feature design, is learned by the
variational encoder. The LSTM decoder looks after temporal information for both
reconstruction and forecasting, generating three-step-ahead DQ predictions from same latent
representations used for anomaly detection. Forecasting and anomaly are handled by a single
learned model of the DQ data distribution, they are not separates systems.
A. Training Objective
The total loss function is:
L = MSE(x̂ , x) + β · Dₖₗ(qϕ(z|x) ∥ p(z)) + λ · MSE(ŷfuture, yfuture),
where β = 0.005 controls the KL divergence term and λ = 0.5 controls the forecasting term.
The reconstruction MSE forces the encoder to preserve the DQ correlation structure within z.
The KL term regularises the posterior toward N(0, I), keeping the latent space smooth and
generalisable. The forecasting term directly supervises three-step-ahead predictions against
the last three timesteps of the input window. The small β prevents regularisation pressure
from overriding reconstruction in the early training epochs, when the encoder has not yet
settled into a meaningful representation.
The reparameterization trick [13] makes the ELBO differentiable end-to-end by expressing
the latent sample as z = μ + σ ⊙ ε, where ε ~ N(0, I) is sampled independently of the model
parameters. This keeps ε outside the gradient graph while gradients flow through μ and σ
normally.
B. Encoder
An encoder has two layers of long-term short-term memory (LSTM) networks (64 hidden
dimensions, 0.1 dropout rate, and batch first), which read in the input sequence of (batch, 24,
7) and produce the final hidden state of the top LSTM layer. Two independent linear
projections map the 64-dimensional final hidden state into a 32-dimensional latent mean μ
and a 32-dimensional logvariance log σ². The 32-dimensional latent vector z, sampled via
reparameterization, captures the full multidimensional DQ correlation of the observed 24-day
window, and its stochasticity ensures that any window that cannot be faithfully modeled by
the encoder within the learned distribution will yield a high reconstruction error, thus
providing an intuitive basis for unsupervised anomaly scoring.
C. Decoder
Reconstruction and Z-conditional Forecasting The decoder is a symmetric two-layer LSTM
(32 input dimensions, 64 hidden dimensions, 0.1 dropout rate) that performs two sequential
operations via weight sharing. During the reconstruction pass, z is replicated for all 24
timesteps and is decoded through the LSTM starting from a zero initial state. This produces
the (batch, 24, 7) shape of a fully reconstructed DQ window. Continuing directly from the
final hidden state of the reconstruction component, the decoder generates the next three time
steps in the forecast pass. Importantly, the forecast passes are based on the sampled z rather
than the previous output when generating the three forecasted time steps.
This z-conditioning resolves a concrete architectural constraint — the decoder input expects
dimension 32 while the decoder output has dimension 7, making direct feedback infeasible —
and is also principled: conditioning every forecast step on the global latent summary of the
full observed window prevents error accumulation across autoregressive steps and anchors
each prediction to the distributional context of the entire 24-day history.