/Department of the Built Environment
/Unit Building Physics and Services
7LY3M0 – Building performance and energy systems simulation
Week [Link] 1
Action: Describe the difference between validation and calibration in the context of
building performance simulation. Are validation and calibration always needed for every
model to be useful? Why (not)?
Student A:
Despite Building Performance Simulation (BPS) tools were initially focused on the design
phase of the building, their usage is now becoming increasingly relevant in post-
construction phases of the building life-cycle (BLC), such as commissioning and
operational management and control.
Defining terms, validation aims to confirm that the predictions made by the simulation
tool provide an accurate representation of how a building system will behave. On the
other hand, calibration aims to minimise discrepancies between measured and simulated
data.
Coakley, D., Raftery, P., & Keane, M. (2014) addresses the importance of model calibration
due to often significant discrepancies between predictions and the actual metered
building energy use, providing indicators to evaluate calibration performance. It also
considers the modelling uncertainty issue, discussing how represented may the real
world be when the system is an arbitrary enclosure of an otherwise open system. Any
model requires certain degree of simplification that allows for validation, and the
modeler main goal is not to collect the richest amount of data to calibrate the model, but
to choose wisely which data until which extent represents more accurately the reality of
the building behaviour without compromising results and/or computational timing for
any given purpose.
Presented by Dirkes II, J.V. & Weaver, B. (2016), often, energy models are wrong because
they fail to accurately represent building performance when considering “accuracy” as
equal to “matching reality exactly”. There are many factors that modellers cannot
consider distinguishing if the discrepancies between measured data and simulation
output are due to poor modelling or external reasons, such as human interaction during
maintenance or inadequate performance of equipment due to manufacturing or
transport complications. These limitations are inherent to the model uncertainty concept
from Coakley, D., Raftery, P., & Keane, M. (2014) but in any case, shall prevent designers
from using BPS. Instead, they become useful to give an insight into what a BPS tool is
1
doing, which is providing an as-accurate-as-possible estimation within a measurable
validating range.
Overall, it can be concluded within the BPS context that validation is always required,
since trusting the model in order to make decisions upon it is its main goal. However,
calibrating may not be compulsory in every case since, depending on the required
accuracy of the modelling, no new data will be needed to validate model behaviour.
Considering new HVAC equipment for a school may not need calibration if the first output
already matches previous energy bills and hence, empirically validates the model.
Otherwise, it is the modeler's task to collect additional new data in order to update the
model, so it fits the validation criteria.
Student B:
Attia et al. (2009) identify three aspects that are of relevance for building performance
simulation (BPS) tools – empirical validation, analytical verification and calibration of
uncertainty.
The American Society of Heating, Ventilating, and Air Conditioning Engineers (ASHRAE)
has published multiple guidelines where they also define calibration. Calibration is the
process whereby the output of the model is compared to real data measured under a
specific set of conditions, with the goal to reduce uncertainty. They further state that the
calibration process includes the identification of parameters which can be adjusted as
well as the acceptable level of uncertainty and inaccuracy (ASHRAE, 2014). The model’s
accuracy is ascertained via uncertainty analysis, which is formally defined as the “process
of determining the degree of confidence in the true value when using measurement
procedures and/or calculations” (ASHRAE 2002; ASHRAE, 2002). It needs to be
mentioned that there is disagreement in literature regarding the approach to uncertainty
analysis for calibration purposes. Growing misunderstanding is caused by the differences
in the criteria chosen for the uncertainty calculations (Ruiz & Bandera, 2017).
The empirical validation is the process that is used to determine the quality of the model
and whether it accurately represents the situation or context it was created for (Attia,
Beltrán, De Herde, & Hensen, 2009).
Validation and calibration can be iterative processes, whereby the information gained
during the validation process can be used to further calibrate the model by identification
of other important parameters that may have been missed previously.
There is a high energy saving potential in the building sector, an estimated 10-40% for
hot climates (Ruiz & Bandera, 2017), which makes the use of BPS attractive.
Technological advances in smart meters, among other things, have resulted in increased
accuracy of models due to more reliable data being accessible to validation. It is
important to recognize that models are only predictive tools and can only be as accurate
as the process of creating them, through calibration, verification and validation, allows
them to be. Whether validation and calibration of a model are necessary depends on its
2
purpose. If a model is to be used as a support for decision making, it should be both
calibrated and validated.
Student A and B:
Given two scientific terms, it has been easy and expected to agree upon their definitions
by both students. More information regarding the calibration process was found than for
validation which highlights the nature of both concepts within BPS. Validation is
subjected to the model's creator’s understanding of the situation and subjective
depending on the model's context. Thus, it is more difficult define universal guidelines to
follow in validation. Calibration, being an analytical task, makes it more feasible to
provide recommendations and steps to take regarding the choice of calibration variables
and how to conduct the process itself.
There is agreement on the need for validation, regardless of the circumstances, as the
model must be validated in order to trust the output. Regarding the calibration process,
student A argues that once a model is aligned with the chosen validation type, there is no
need for calibration, whereas student B claims that calibration increases the accuracy of
the model's output. These two opinions do not have to be exclusive. Both students agree
that the difficulty lies in analysing the purpose of the simulation and deciding upon an
acceptable degree of accuracy. In real life applications, these decisions are also influenced
by the budget and time allocated to the modelling and simulation process. Thus, the
calibration process should not be overengineered and seen in the context of the
simulation's purpose.
References:
American Society of Heating Refrigerating and Air-Conditioning Engineers (ASHRAE).
(2002). ASHRAE Guideline 14-2002 Measurement of Energy and Demand Savings.
Ashrae, 8400, 170.
American Society of Heating Ventilating and Air-Conditioning Engineers (ASHRAE). (2014).
ASHRAE Guideline 14-2014, Measurement of Energy and Demand Savings. Ashrae.
Attia, S., Beltrán, L., De Herde, A., & Hensen, J. (2009). “Architect friendly”: A comparison of
ten different building performance simulation tools. IBPSA 2009 - International
Building Performance Simulation Association 2009, 204–211.
Ruiz, G. R., & Bandera, C. F. (2017). Validation of calibrated energy models: Common errors.
Energies, 10(10). [Link]
3
(Coakley et al., 2014) A review of methods to match building energy simulation models to
measured data. Renewable and sustainable energy reviews, 37, 123-141.
(Dirkes II & J.V. & Weaver, 2016) Modeled Performance Isn't Actual
Performance. ASHRAE Journal 58.5