I NFORMATION ON DMU(DATA M ODEL U TILITY )

JiYingBo 2411874

December 16, 2024

A BSTRACT
The multifaceted concept of information plays a central role in various fields such as human knowl-
edge, communication, and technology. This article demonstrates the importance of information as a
quantitative tool in scientific research by analyzing its applications in data, models, and utility. In
the section on information measurement utility, the focus is on dealing with uncertainty in real-life
situations, and fuzzy sets are introduced as a mathematical tool to handle uncertainty. In the section
on information measurement models, we introduce entropy to measure model training loss, such as
cross-entropy, mutual information, and conditional entropy, along with their applications in machine
learning and information theory. Finally, the section on information measurement data emphasizes
the importance of feature distribution quantification and the information bottleneck in transforming
data into meaningful information, which is crucial for enhancing the learning performance of models.

1 Information
The concept of information holds fundamental and pervasive significance within the realm of human knowledge. At
its core, information can be defined as the interpretation of perceptible or abstract phenomena, serving as a means to
capture the essence of both tangible and intangible entities. It is important to note that any natural process or observable
pattern, deviating from complete randomness, inherently carries a certain measure of information. This universal notion
applies regardless of the medium through which the information is conveyed.
In a general sense, information refers to the content conveyed through communication or transmission, taking various
forms such as language, images, sound, symbols, and more. It represents meaningful data transmitted to a receiver
with the purpose of increasing knowledge, understanding, or influencing behavior. Claude Shannon proposed that
"information is something used to eliminate random uncertainty" and introduced the concept of information entropy to
quantify the amount of information [1]. Norbert Wiener, the founder of cybernetics and a renowned mathematician,
described information as follows: "Information is information, not matter or energy. Any materialism that fails to
recognize this cannot survive in our times."
In essence, the concept of information transcends boundaries and finds relevance across diverse domains. It provides
a universal framework for comprehending and describing the underlying structure and meaning inherent in our
surroundings. Whether it be the intricate patterns observed in biological systems, the organized sequences of symbols
in human language, or the encoded representations in digital technology, information serves as a unifying principle
that underlies the fabric of our understanding. With the advancement of information technology, Claude Shannon
provided a more substantial explanation of the concept of information within the field of information theory. Shannon’s
groundbreaking work revolutionized our understanding of information by introducing quantitative measures and
rigorous mathematical foundations.
In Shannon’s framework, information is not merely seen as a subjective interpretation but rather as a measurable entity.
He introduced the notion of "bits" as a unit for measuring information, representing the fundamental building blocks of
digital communication. Shannon’s theory established a framework for quantifying the amount of information contained
in a message, taking into account factors such as probability and redundancy. Moreover, Shannon’s work highlighted
the concept of entropy, which measures the uncertainty or randomness within a system. By associating entropy with the
amount of information, Shannon provided a deeper understanding of how information is encoded, transmitted, and
received within a noisy channel. The contributions of Shannon and the development of information theory have had a
profound impact on various fields, including telecommunications, data compression, cryptography, and more. This
 A PREPRINT - D ECEMBER 16, 2024

formalization of information as a measurable and quantifiable entity has paved the way for the efficient processing,
storage, and transmission of information in the digital age.
In the context of the continuous development of artificial intelligence technologies, data-driven machine learning has
gradually become the mainstream approach. However, it appears that the concept of information within this paradigm
has been overlooked, resulting in a lack of theoretical foundations rooted in information theory.

2 Information to Utility

2.1 Uncertainty

In real-life situations, information is often fraught with uncertainty. When a dataset is insufficient to support precise
computation of event outcomes by a model, it becomes necessary to employ mathematical tools to enhance the
computational efficiency of the model. Quantifying uncertainty is crucial for assessing the reliability and robustness of
a model. By quantifying the uncertainty associated with the model’s predictions, we can identify potential errors and
biases that may arise in specific scenarios. This aids in identifying situations where the model may be unreliable or
exhibit higher error rates, enabling improvements or the implementation of appropriate measures to enhance model
utility.

2.2 Definition of Uncertainty

Economist Frank Knight defined uncertainty as "people’s inability to forecast the likelihood of events happening" [2].
When we encounter an event or situation, uncertainty exists when we lack precise understanding of its outcomes or the
likelihood of its occurrence.

2.3 Classification of Uncertainty

Uncertainty can be classified into four main categories [3]:

1) Metrical: This type of uncertainty arises from imprecision in measurement. It occurs when there are limitations or
errors in the measurements or data used to analyze a situation.
2) Structural: Structural uncertainty arises from the complexity of certain issues or systems. Complex systems can
involve multiple variables, interactions, and uncertainties, making it difficult to predict outcomes accurately. The
interdependencies and nonlinear relationships within such systems contribute to structural uncertainty.
3) Temporal: Temporal uncertainty refers to the inherent unpredictability of future or past events or phenomena. Some
events or phenomena are inherently random in nature, and their outcomes cannot be completely determined or predicted
with certainty.
4) Translational: Translational uncertainty arises from different conceptions or interpretations of uncertain results. It
occurs when there are variations in the understanding or communication of uncertain information among different
individuals, groups, or disciplines.

2.4 Mathematical Tool for Studying Uncertainty - Fuzzy Sets

In 1965, Lotfi A. Zadeh introduced the concept of fuzzy sets. "A fuzzy set is a class of objects with a continuum of
grades of membership. Such a set is characterized by a membership (characteristic) function which assigns to each
object a grade of membership ranging between zero and one" [4].
Unlike traditional set theory, where an element either belongs to a set or does not belong to it without any intermediate
state, fuzzy sets use membership functions to describe the degree of membership of an element in a set. The membership
function can take values between 0 and 1, representing the degree of membership of the element in the set, with higher
values indicating a higher degree of membership.
In general, fuzzy sets can be represented as follows:

A(x1 ) A(x2 ) A(xn )

A= + + ..... + (1)
x1 x2 xn
A(xi ) represents the membership degree of element xi in the set A.

2
 A PREPRINT - D ECEMBER 16, 2024

At the theoretical level, fuzzy sets find applications in various areas, including fuzzy logic, approximate reasoning,
paraconsistent reasoning, fuzzy rule-based systems (such as fuzzy expert systems and fuzzy control), fuzzy data mining,
fuzzy optimization, fuzzy decision-making, and more [5].
Fuzzy sets have wide-ranging applications in various fields, providing solutions to handle uncertainty and fuzziness in
problems related to reasoning, decision-making, and optimization.

3 Information to Model
3.1 Information Measurement

In machine learning, information gain is frequently utilized to optimize and determine the structure of models.
Information entropy is employed as a loss function to measure the error in the model’s outcome predictions. Entropy
serves as a tool for converting information into a format that the model can assimilate.

3.2 Entropy

In physics, entropy is a concept that describes the level of disorder in a system. The definition of entropy can be
explained from the perspectives of thermodynamics and statistical physics.
In thermodynamics, entropy can be understood as a measure of the variability of a system’s states. It is directly
proportional to the logarithm of the number of microstates accessible to the system. The definition of entropy in
thermodynamics is based on the second law, which states that in an isolated system, entropy always increases or remains
constant and never decreases. For a reversible process, the change in entropy is zero, while for an irreversible process,
the change in entropy is positive.
In statistical physics, entropy can be defined based on the microscopic state distribution of a statistical system. If a
system has multiple microstates with different probabilities of occurrence, the entropy of the system can be defined as
the negative weighted average of the logarithm of the probabilities of all possible states. This definition is known as
statistical entropy or Gibbs entropy.
Both thermodynamic entropy and statistical entropy serve as measures of the degree of disorder in a system. When the
microscopic states of a system are more ordered or concentrated, the entropy is lower, whereas when the microscopic
states are more disordered or dispersed, the entropy is higher. Furthermore, entropy has wide applications in information
theory.
From the perspective of academic research, this concept can be extended to the aspect of information measurement.
In information theory, entropy can be used to quantify the uncertainty or randomness in a set of information. It
provides a measure of the average amount of information required to describe or represent an event or a message.
High entropy indicates a greater level of uncertainty or unpredictability, while low entropy signifies a higher level of
order or predictability in the information. Thus, entropy serves as a fundamental tool for quantifying and analyzing
the information content and structure in various fields, including communication systems, data compression, and
cryptography.

3.3 Information Entropy

Shannon introduced the concept of information entropy in information theory, which is used to measure the uncertainty
or average amount of information in a random variable.
Information entropy is calculated based on the probability distribution of a random variable. For a discrete random
variable X, its information entropy H(X) can be calculated using the following formula:
N
X
H=− pk log2 pk (2)
k=1

The information entropy is typically measured in bits and represents the amount of information or distinguishable states.
Higher entropy indicates greater uncertainty in the random variable and requires more information for its description.
Conversely, lower entropy signifies reduced uncertainty and correspondingly decreased information.
When all possible values of a random variable have equal probabilities, the entropy reaches its maximum value,
indicating maximum uncertainty. Conversely, when certain values of the random variable have higher probabilities, the
entropy decreases, indicating reduced uncertainty.

3
 A PREPRINT - D ECEMBER 16, 2024

Information entropy finds wide applications in information theory and communication. It can be used to measure the
efficiency of data compression, where higher entropy implies greater difficulty in achieving effective compression.
Additionally, information entropy is relevant to cryptography, as it helps assess the security of cryptographic algorithms.
In data communication, information entropy can be used to measure the capacity of a transmission channel, i.e., the
maximum amount of information the channel can transmit.
In conclusion, information entropy is a crucial concept that provides a measure of uncertainty and information content
in a random variable. It holds significant importance in understanding the nature of information and its applications in
various information processing and communication systems.

3.4 Measures of multiple distributions or variables

Cross-entropy The cross-entropy is a measure of information between two probability distributions, denoted as p and
q. The cross-entropy of p relative to q is defined as follows:

H(p, q) = H(p) + KL(p||q) (3)

Here, H(p) represents the entropy of p, and KL(p||q) is the Kullback-Leibler divergence of p from q.
For discrete probability distributions p and q, the cross-entropy can be expressed as:
X
H(p, q) = − p(x) log q(x) (4)

The concept of cross-entropy was initially introduced in algorithms for optimizing computer simulation models with
rare events [6]. However, it was soon recognized as an important mathematical tool. With the rapid development of
machine learning, cross-entropy has been widely used as a loss function.
In recent research, Malik Boudiaf has proven that cross-entropy is an upper bound for a new class of pairwise loss
functions, which share the structure of minimizing intra-class distances while maximizing inter-class distances [7]. Anqi
Mao et al. have provided theoretical analysis for a broad class of loss functions, including cross-entropy, generalized
cross-entropy, and other similar loss functions. They have also established the consistency bounds for these loss
functions [8].

Mutual Information Mutual information is defined as follows:

XX p(x, y)
I(X; Y ) = H(X) − H(X|Y ) = p(x, y) log (5)
p(x)p(y)
x∈X y∈Y

The concept of mutual information was first introduced by Shannon, and it represents the degree of correlation between
two pieces of information [1].
In general, estimating the true mutual information is challenging. Pengyu Cheng proposed a novel upper bound for
mutual information called Contrastive Log-ratio Upper Bound (CLUB) [9]:

ICLU B (X; Y ) = Ep(x,y) [log p(y|x)] − Ep(x) Ep(y) [log p(y|x)] (6)

Building upon the upper bound of mutual information, Chen Zhao proposed a lower bound called MI lower bound to
approximate the true mutual information entropy [10].
Mutual information has also found wide applications in various fields of deep learning. For example, R. Steuer used
mutual information to cluster genes [11]. Huiqiang Xie computed the mutual information between transmitted symbol
x and received symbol y in semantic transmission, providing additional information for receiver training [12].

Conditional Entropy Conditional entropy, based on X given Y , is defined as follows:

X p(x)
H(Y |X) = p(x, y) log (7)
p(x, y)
x∈X,y∈Y

It represents the amount of information entropy in random variable Y when random variable X is known or given.

4
 A PREPRINT - D ECEMBER 16, 2024

In machine learning, models are susceptible to adversarial examples, calibration errors, and overfitting to random labels
in the dataset. To address these challenges in robust generalization, Ian Fischer proposed the concept of conditional
entropy bottleneck [13]:

CEB ≡ min I(X; Z|Y ) − γI(Y ; Z) (8)

Z

Here, I(X; Z|Y ) measures the mutual information between X and Z conditioned on Y, and I(Y ; Z) measures the
mutual information between Y and Z. By minimizing the conditional mutual information while maximizing the mutual
information between Y and Z, the conditional entropy bottleneck aims to improve the robustness and generalization of
machine learning models.

4 Information to Data
Data contains a wealth of information, and it is common practice to abstract data into features to represent this
information. The process of feature extraction and measuring the information content of features is a crucial step in
transforming data into meaningful information.

4.1 Quantifying Feature Distribution

In practical terms, the representation of data is mostly composed of features, and measuring the amount of information
in a feature becomes a challenging problem. Paper [14] provides a specific method for quantifying features as follows.
First, consider a multi-classification task X → Y = {1, . . . , K}, where X , Y, and K represent the input space, output
space, and the number of categories respectively. Then, use the symbol ϕ to denote a feature. Specifically, assume
that the classifier f can be decomposed as g ◦ h, where g ∈ G : Rd → RK is the top classifier and h : X → Rd is a
d-dimensional feature extractor, i.e.,.
h(x) = (ϕ1 (x), ϕ2 (x), . . . , ϕd (x))T , ϕi ∈ Φ,
where Φ is the set of scalar feature maps which map X to R and d is fixed, each ϕi ∈ Φ is referred to as a feature. Given
a domain e in a domain set E, denote the d-dimensional random vector h(X e ) as he , the one-dimensional feature ϕ(X e )
as ϕe , and the conditional distribution of he , ϕe given Y e = y as P(he |y), P(ϕe |y). Assume the data distribution is
1
balanced in every domain, i.e., P (Y e = y) = K , ∀y ∈ Y, e ∈ E. The definition of “informativeness” for the feature ϕ
is given as follows.
Definition 1. (Informativeness) [14] The informativeness of feature ϕ(·) across a domain set E is
1 X
Iρ (ϕ, E) = min ρ(P(ϕe |y), P(ϕe |y ′ )), (9)
K(K − 1) ′
e∈E
y̸=y ∈Y
y,y ′ ∈E

where ρ(·, ·) be a symmetric “distance” between two distributions.

Note that ρ can have many choices, such as L2 Distance, Total Variation, and symmetric KL-divergence, etc.
From equation 9, it can be seen that the meaning of informativeness can be understood as the discriminative power of
different classifications of X under the feature mapping ϕ for a given domain e. The larger the difference between the
two distributions, the higher the informativeness, indicating that the information is more useful. Specifically, when the
“distance” ρ = 0, P (Y e = y) = 1/K, which is equivalent to random classification, making the information ineffective.

4.2 Information bottleneck

Once information is extracted from the data, we require an information bottleneck to distill the information, allowing it
to have the maximum impact on the model.
The Information Bottleneck Theory is an information theory framework used to understand and explain the information
processing process in tasks such as data compression, pattern recognition, and machine learning. This theory was
initially proposed by Naftali Tishby and others to address the question of how to extract useful information from input
data during the learning process [15]. The core idea of the Information Bottleneck Theory is to achieve data compression
and efficient representation by minimizing the redundancy between the output and the input while preserving the
essential information.

5
 A PREPRINT - D ECEMBER 16, 2024

The concept of Information Bottleneck (IB) [15] arises as a significant paradigm in the field of data-driven modeling.
It centers around the idea of finding a balance between preserving relevant information from the input data while
discarding redundant or irrelevant details. The IB principle, rooted in information theory, aims to distill the most salient
features from the data, enabling models to capture essential patterns without becoming overly complex. By minimizing
the mutual information between the input and output while maximizing the mutual information between the output and
the target, the IB framework facilitates the creation of more concise and efficient models, enhancing their generalization
capabilities across various tasks and datasets.
An information bottleneck is a compression scheme for extracting from a random variable V all the information
relevant for the estimation of another random variable W while discarding all irrelevant information in V . Information
bottlenecks have been put forward as a theory of how implicit feature representations evolve during training [16]. That
line of work studies how the prediction process transforms information from a test datum during prediction (i.e. as one
moves through layers of a neural network), and is thus distinct from our study of how learning algorithms are able to
extract information from training data sets.
The proposed framework for behavior computing is built upon the foundation of a behavior structure model. The
behavioral structure model intricately models data as behavior structures, typically represented in a graph format. The
diversity and quality of data directly impact the performance of machine learning models. From an informational
perspective, the quality of data is essentially equivalent to the size of information. We denote the information carried by
any data as δ, where the information κ learned by a machine learning model from the initial data must be less than δ.
This underscores the limitation on the model’s learning imposed by the information contained in the initial data.
In the real world, changes follow certain patterns, and various aspects of the world exhibit internal connections.
Therefore, data extracted from the real world is not merely a series of bit signals from data records; rather, it reflects
deeper structures and patterns. This viewpoint provides us with the opportunity to gain a deeper understanding of the
data.
In the process of constructing a fine-grained behavioral structure model, we can leverage knowledge, such as establishing
connections between data samples to increase information gain or optimizing data features to reduce noise during the
model learning process. This emphasizes two key aspects in building a behavioral structure model: the application
of knowledge and the optimization of data features. These methods aim to enhance the model’s understanding and
learning capabilities of the data.
Ultimately, we emphasize the importance of analyzing the effectiveness of the behavioral structure model from an
information perspective. This implies a focus on how to better comprehend and utilize the information within the data
to improve the model’s performance.

5 Discussion

This article thoroughly examines the multifaceted concept of information and its pivotal role in various fields, including
human knowledge, communication, and technology. We highlight the significance of information as a means to mitigate
uncertainty and the concept of quantifying information through information entropy. Initially, we discuss the process of
transforming information into utility, with a particular focus on the quantification and management of uncertainty in
real-world scenarios. Uncertainty is categorized into four types: metrological, structural, temporal, and translational,
and fuzzy sets are introduced as a mathematical tool to address uncertainty.
Subsequently, we delve deeper into the measurement of information within machine learning models, emphasizing
the use of entropy as a loss function to assess prediction errors. The article distinguishes between thermodynamic
entropy and statistical physics entropy, as well as their interplay with information theory, especially in quantifying the
uncertainty within information sets. Information entropy is underscored as a critical tool for evaluating the efficiency of
data compression, the security of cryptographic measures, and the capacity of communication channels.
Finally, we emphasize the importance of quantifying feature distributions and the information bottleneck in converting
data into meaningful information, which is essential for enhancing model performance.
In future research, we aim to achieve more precise calculations of fuzzy sets and introduce new forms of information
entropy to measure uncertainty. Ultimately, we seek to advance the development of large models by integrating
information to connect data, utility, and models into a cohesive whole.

6
 A PREPRINT - D ECEMBER 16, 2024

References
[1] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–
423, 1948.
[2] Frank Knight. Risk, Uncertainty and Profit. University of Chicago Press, 1921.
[3] William D Rowe. Understanding uncertainty. Risk analysis, 14(5):743–750, 1994.
[4] Lotfi A Zadeh. Fuzzy sets. Information and control, 8(3):338–353, 1965.
[5] H-J Zimmermann. Fuzzy set theory. Wiley interdisciplinary reviews: computational statistics, 2(3):317–332,
2010.
[6] Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of
Operational Research, 99(1):89–112, 1997.
[7] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben
Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European
conference on computer vision, pages 548–564. Springer, 2020.
[8] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications.
arXiv preprint arXiv:2304.07288, 2023.
[9] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive
log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779–1788.
PMLR, 2020.
[10] Chen Zhao, Le Wu, Pengyang Shao, Kun Zhang, Richang Hong, and Meng Wang. Fair representation learning
for recommendation: a mutual information perspective. In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 37, pages 4911–4919, 2023.
[11] Ralf Steuer, Jürgen Kurths, Carsten O Daub, Janko Weise, and Joachim Selbig. The mutual information: detecting
and evaluating dependencies between variables. Bioinformatics, 18(suppl_2):S231–S240, 2002.
[12] Huiqiang Xie, Zhijin Qin, Geoffrey Ye Li, and Biing-Hwang Juang. Deep learning enabled semantic communica-
tion systems. IEEE Transactions on Signal Processing, 69:2663–2675, 2021.
[13] Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[14] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical
framework of out-of-distribution generalization. Advances in Neural Information Processing Systems, 34:23519–
23531, 2021.
[15] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint
physics/0004057, 2000.
[16] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee
information theory workshop (itw), pages 1–5. IEEE, 2015.