KNOWLEDGE DISCOVERY TOOL

1. Q-Matrix
Assessing students' knowledge based on their test responses has led to the development of the Q-
Matrix. This tool presents the precise relationship between test questions and the underlying
concepts or attributes. In the form of a binary matrix, this domain-independent knowledge model
displays the connections between test items and the concepts necessary to comprehend them. The Q-
Matrix is an M x N matrix, where M represents the number of assessment questions, and N represents
the total number of concepts required to understand all questions. The matrix element A[i, j] equals 1
if the ith concept is answered correctly for the jth question and 0 if not. The matrix values are not
limited to {0, 1}, but can also be real numbers in the range [0,1], representing the probability that a
student who knows the ith concept will correctly answer the jth question.

Table 1: Example Q-matrix

Understanding matrix values and implications

In Table 1, each row corresponds to a unique skill, and each column corresponds to a unique question.
Each value in the table, denoted by Q (c, q), represents the probability of a student answering the
question q correctly, given that he or she has mastered all other skills in the table except skill c. The
interpretability of each value in a q-matrix is straightforward; it is either 0 or 1. An example of a binary
q-matrix is given in Table 1. A q-matrix contains a one if a question is related to the concept and a
zero if not. For example, in this q-matrix, questions q1 and q5 are both related by concept c1, while q1
is also related to q2 and q4 by concept con2.
Due to various knowledge and assessment characteristics, students' responses rarely match ideal
responses generated from the matrix. Still, by assigning the closest ideal response to a student's
response vector, it can be assumed which concepts the student does and which he does not know.
This information can be used to direct him in further learning.

Q-Matrix Algorithm
The q-matrix algorithm is a simple hill-climbing algorithm that creates a matrix representing
relationships between concepts and questions directly from student response data. The algorithm
varies c, the number of concepts, and the values in the q-matrix, minimising the total error for all

Dr. Saswati Mukherjee, School of Education Technology, JU 1

students for a given set of n questions. Each hill-climbing search is seeded with different random q-
matrices to avoid local minima, and the best of these is kept.
Initially, the number of concepts, denoted as 'c', is set to one, and a random Q-matrix of concepts
versus questions is generated with values ranging from 0 to 1. The response data of students are
grouped according to "concept states," and the total error associated with the students assigned to
the concept states is calculated. After the error is computed, each value in the Q-matrix is adjusted
slightly, and if the overall Q-matrix error improves, the change is saved. This process is repeated for all
values in the Q-matrix several times until the error stabilises. Once a Q-matrix is computed, the
algorithm is run again with a new random starting point several times, and the Q-matrix with the
lowest error is saved to prevent settling for a suboptimal solution. Although not guaranteed to be the
absolute minimum, this approach yields an acceptable Q-matrix for a given number of concepts. To
determine the best number of concepts to use in the Q-matrix, the algorithm is repeated to increase
the values of 'c'. The final Q-matrix is selected when adding an additional concept, which does not
significantly reduce the overall Q-matrix error, and the number of concepts is substantially smaller
than the number of questions.

2. Knowledge Tracing
Knowledge Tracing (KT) is a method used to measure a student's knowledge based on their
performance during learning. KT helps determine how to effectively track students' learning progress
through online interaction with teaching materials. It aims to observe, represent, and quantify a
student’s knowledge state, e.g., the mastery level of skills underlying the teaching materials. This is an
important task in developing modern intelligent tutoring systems as it aims to monitor the evolution
of a student's knowledge over time. KT is a supervised sequence learning task based on a student's
previous interactions with an exercise (X = x1, x2, ..., xt). The goal is to predict the probability that the
student will answer the next question of the exercise correctly, denoted as P(r t = 1|et+1, X). The
interactions are shown as xt = (et, rt), where et is the exercise attempted by the student at time t and rt
is the correctness of the student’s answer.
BKT or Bayesian Knowledge Tracing, is a probabilistic model used in educational data mining and
learning analytics. BKT aims to model the probability that a learner possesses knowledge of each skill
as they progress in their learning. The goal of BKT is to estimate the knowledge states of individual
learners over time based on their interactions with educational tasks or assessments.
In BKT, the learner's knowledge is considered a latent variable, while their responses to items (i.e.,
their performance) are treated as observable variables. The knowledge is represented as a binary
variable, indicating whether the learner has learned the skill. Additionally, the items are scored
dichotomously, with answers categorized as either correct or incorrect.
BKT leverages Bayesian inference to update the estimates of a learner’s knowledge state at a
particular time step t as the learner responds to questions or completes learning activities (𝑜𝑏𝑠𝑡).
The model includes two knowledge parameters for each learner: the probability of knowing a concept
before encountering a task (i.e., prior knowledge; 𝑃(𝐿0)), and the probability of learning or acquiring

Dr. Saswati Mukherjee, School of Education Technology, JU 2

knowledge from the task at time step t (i.e., learned knowledge; 𝑃(𝐿𝑡)). The probability of
transitioning from the not-known state to the known state after each answer is denoted as 𝑃(𝑇).
In addition to knowledge parameters, the model also involves two performance parameters: the
probability of making a mistake when applying a known skill (slip; 𝑃(𝑆)) and the probability of
answering an item correctly with a not-known skill (guess; 𝑃(𝐺)).
These four parameters in BKT are utilized to update the learning probability, representing the
likelihood of the learner’s knowledge of a specific skill. More specifically, as the learner responds to
the items, BKT updates 𝑃(𝐿𝑡) based on the accuracy of their response (correct or incorrect):

𝑃(𝐿𝑡 )(1 − 𝑃(𝑆))

𝑃(𝐿𝑡 |𝑜𝑏𝑠𝑡 = 1) = − − − 𝐸𝑞. 1
𝑃(𝐿𝑡 )(1 − 𝑃(𝑆)) + (1 − 𝑃(𝐿𝑡 ))𝑃(𝐺)
𝑃(𝐿𝑡 )𝑃(𝑆)
𝑃(𝐿𝑡 |𝑜𝑏𝑠𝑡 = 0) = − − − 𝐸𝑞. 2
𝑃(𝐿𝑡 )𝑃(𝑆) + (1 − 𝑃(𝐿𝑡 ))(1 − 𝑃(𝐺))

As the learner transitions from one step (t) to the next (𝑡+1), the updated prior for the following time
step can be calculated by Equation 3.
𝑃(𝐿𝑡+1 ) = 𝑃(𝐿𝑡 |𝑜𝑏𝑠𝑡 ) + (1 − 𝑃(𝐿𝑡 |𝑜𝑏𝑠𝑡 ))𝑃(𝑇) − − − 𝐸𝑞. 3
which suggests that the learner is likely to transition from a not-known state to a known state by
learning from immediate feedback and any other instructional support; however, the probability of
forgetting 𝑃(𝐹) remains zero. Figure 1 illustrates the workflow of the standard BKT model.

Figure 1: Workflow of BKT

Dr. Saswati Mukherjee, School of Education Technology, JU 3

3. Factor analysis
Factor analysis is a statistical method used to uncover the underlying structure of a set of variables. It
helps reduce the number of variables to smaller factors, simplifying a complex dataset. Factor analysis
can be used for several purposes:
• To reduce many variables for modelling when individual modelling of all measures is impractical.
• To select a subset of variables from a large set based on their correlations with the principal
component factors.
• To create a set of uncorrelated variables to handle multicollinearity in regression analysis.
Researchers use factor analysis when they suspect that latent factors cause observable variables to
covary. This method helps identify these hidden variables. Factor analysis simplifies complex datasets
by reducing observed variables to a smaller set of unobserved factors, helping to understand the
underlying reality of the subject area. The basic premises of factor analysis include:

1. Latent variables influence observed variables.

2. Inter-individual differences in observed variables are due to latent variables and measurement
errors.
3. Each type of factor and measurement error contributes to a portion of variance.

Figure 4: Variance structure of observed

Figure 3: Elements that influence observed
variables
variables

Figure 3 shows three things (common factors, specific factors, and measurement error) that
influence the observed variables. Firstly, the common factors/latent variables, which generate
more than one of the observed variables (e.g., “math ability” might give rise to “addition test
score,” “multiplication test score,” and “division test score”). Secondly, the specific factors/latent
variables generate only one of the observed variables (a common factor can become a specific
factor if all factors are removed. However, there must be at least one of the observed variables
that is generated). Thirdly, what influences the observed variables is the measurement error,
which is not latent but is often due to unsystematic events that influence measurement.
Measurement error is closely tied to reliability. Figure 4 shows that the variance of a given
observed variable is due to factors that influence other observed variables, factors that influence

Dr. Saswati Mukherjee, School of Education Technology, JU 4

 only the given observed variable, and measurement error. Common variance is sometimes
referred to as ‘commonality’, and specific variance and error variance are often combined and
referred to as ‘uniqueness’.
For example, factor analysis of students’ academic performance is an exploratory multivariate
analysis that either reduces the number of variables in a model or detects the correlation among
the variables. Many studies have been conducted to identify factors such as learning style, age,
gender, teacher factor, location, economic status of parents, and many others that affect students'
academic performance in primary school, college, and even at the university level. It regroups
variables into a limited set of clusters based on shared variance. In factor analysis, observed
variables are represented as a linear combination of the unobserved factors.
While all factor analysis aims to find latent factors, researchers use it for two reasons. They either
want to explore and discover the structure within a dataset or to confirm the validity of existing
hypotheses and measurement instruments. There are two main types of factor analysis:
exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).
• Exploratory factor analysis (EFA) is data-driven; here, the data collected determines the resulting
factors. This analysis tries to uncover complex patterns by exploring the dataset and testing
predictions.
• Confirmatory factor analysis (CFA) tests factors developed a priori. It is seen as a process for
testing previous knowledge. It attempts to confirm hypotheses and uses path analysis diagrams to
represent variables and factors. CFA is also useful in checking a measurement tool's reliability with
a new population of subjects or further refining an existing instrument.

Condition for the Use of Factor Analysis

Factor analysis is typically used with ordinal or continuous variables but can also be applied to
categorical and dichotomous variables. The data for factor analysis should be normal, univariate, and
multivariate.
• When computing the correlations, it is assumed that there is a linear relationship between the
factors and the variable. A factor should have at least three variables to be considered as such,
although this can depend on the study's design. A factor with two variables is only considered
reliable when the variables are highly correlated (r > 0.70), and factors with two or fewer
variables should be interpreted with caution.
• The sample size for factor analysis should be at least 30, and each factor should have at least 5
to 10 observations.
• It is advisable to use a heterogeneous sample rather than a homogeneous sample, as
homogeneous samples reduce the variance and factor loadings. Additionally, the correlation
between the variables should be 0.30 or greater, as anything lower represents a weak
relationship between the variables.
• The dataset should not contain multicollinearity or singularity. Variables with singularity and
multicollinearity issues should be removed from the dataset.

Dr. Saswati Mukherjee, School of Education Technology, JU 5

Factor analysis aims to reduce the number of variables and interpret the results. This can be
achieved in two steps:
- Factor extraction
- Factor rotation
Factor Extraction
Factor extraction is a mathematical method used to identify factors within a dataset. The model type
and the number of factors to be identified are determined during factor extraction. Factors are
derived from correlation matrices by transforming these matrices using eigenvectors. An eigenvector
of a square matrix is a vector that, when multiplied by the square matrix, results in a vector that is a
multiple of the original vector. This multiple is known as the eigenvalue, which indicates the variance
accounted for by each factor. Each extracted factor will have an eigenvalue, with the first factor
aiming to capture as much variance as possible, leading to subsequent eigenvalues being lower than
the first. Eigenvalues exceeding 1 are considered "stable." The sum of all eigenvalues is equal to the
number of observed variables in the model. These eigenvalues are then assigned to factors based on
the amount of explained variance. Factor analysis software often produces scree plots (Figure 5)
displaying the variance explained by each factor. The 'cut off' occurs just before the 'bend' in the
scree plot - typically around 2 or 3 factors in Figure 5. Eigenvalues and scree plots aid in identifying
the optimal number of factors for the selected data.

Figure 5: Scree Plot

Factor Loading
Factor loading gives an idea about how much the variable has contributed to the factor; the larger the
factor loading, the more the variable has contributed to the factor. Factor loading is very similar to
weights in multiple regression analysis, and it represents the strength of the correlation between the
variable and the factor. It is the correlation coefficient for the variable and factor, and the values range
from -1 to +1. The sign indicates the direction of the relations (positive or negative), while the absolute
value indicates the strength. Stronger relationships have factor loadings closer to -1 and +1. Weaker
relationships are close to zero. Higher factor loading represents that the factor extracts sufficient
variance from that variable. A rule of thumb is to consider loadings above 0.3.
Eigenvalues – Eigenvalues are a good criterion for determining a factor. Eigenvalues are also called
characteristic roots. Eigenvalues show variance explained by that particular factor out of the total
variance. From the commonality column, it can be known how much variance is explained by the first

Dr. Saswati Mukherjee, School of Education Technology, JU 6

factor out of the total variance. For example, if the first factor explains 68% of the variance out of
the total, the other factor will explain 32%. If Eigenvalue is greater than one, that factor is considered;
if Eigenvalue is less than one, then that factor is not considered.

Factor Rotation
Factor rotation (Figure 6) occurs after extracting the factors to achieve a simpler structure and
improve interpretability. The purpose of rotating factors is to make the results easier to interpret, as
unrotated factors can be confusing. The goal of rotation is to create a simple structure where each
variable loads heavily on just a few factors, making interpretation easier. This simple structure allows
each factor to define a distinct cluster of related variables, simplifying interpretation. There are two
types of rotation: orthogonal (perpendicular), where factors are not allowed to be correlated with
each other, and oblique, where factors can be correlated (Figure 7). Examples of orthogonal rotation
include varimax, quartimax, and equamax. Examples of oblique rotation include oblimin, promax, and
geomin (Figure 8). After rotation, the factors are rearranged to go through clusters of shared
variances, making the factors easier to interpret.

Figure

Figure 6: Variable loading plot

Figure 7: Correlation between variables

Dr. Saswati Mukherjee, School of Education Technology, JU 7

 Figure 8: Factor rotation types

Advantages of Factor Analysis:

• Can be used for both objective and subjective attributes.
• Identifies hidden dimensions or constraints.
• Cost-effective and provides accurate results.
Disadvantages of Factor Analysis
• Depends on the researcher's ability to develop a complete set of product attributes.
• Naming the factors can be difficult.
• Cannot produce a meaningful pattern if the observed variables are unrelated.
• Requires theoretical analogy to understand the justification behind the factors.

Dr. Saswati Mukherjee, School of Education Technology, JU 8