DATA ANALYTICS FOR FINANCE
STRATEGIC LEVEL EXAMINATION
SAMPLE PAPER 2021
NOTES:
You are required to answer all 5 Questions.
TIME ALLOWED:
3.5 hours, plus 10 minutes to read the paper.
Examination Format
This is an open book examination. Hard copy material may be consulted during this
examination, subject to the limitations advised on the Institute’s website.
Marks
Marks for each question are shown. The pass mark required is 50% in total over the whole
paper.
Answers
You are reminded that candidates are expected to pay particular attention to their
communication skills and care must be taken regarding the format and literacy of the solutions.
The marking system will take into account the content of your answers and the extent to which
answers are supported with relevant legislation, case law or examples, where appropriate.
DATA ANALYTICS FOR FINANCE
STRATEGIC LEVEL EXAMINATION – SAMPLE PAPER 2021
Data Science Specialists Ltd. provide statistical consulting to companies across Europe. The
company specialises in analysis, which will support businesses in gaining an understanding
from their data. It also provides forecasting services to support decision makers. The company
has significant expertise in areas such as machine learning, artificial intelligence, data mining
and deep learning. The company also offers training courses that upskill people in the areas
of statistics, computer programming, data science and analytics, in both online and face to
face settings (subject to public health advice).
The CEO of Data Science Specialists Ltd. is Damian O’Sullivan. His vision for the organisation
is “to be the leader in statistical consulting and training across Europe”. The company has just
successfully won a contract to perform analyses and support the development of a data
science programme within a large biopharmaceutical company.
Damian views this contract as an opportunity to gain a foothold in this booming sector. To this
end, there are a number of key questions that Damian needs answered to a high standard.
He believes that if the analyses are delivered well, then the likelihood of repeat business is
very high.
The client has provided data on numerous areas of the business and Damian aims to provide
key support using expertise in:
• Linear regression,
• Decision trees,
• Random forests and other machine learning models,
• Basic descriptive statistics,
• Data systems architecture and strategies for data analytics programmes to yield
maximum benefit,
• Time series analysis,
• Interpreting output from statistical software packages,
• Data protection and data integrity.
In applying these techniques, Data Science Specialists Ltd. have helped numerous clients to:
• Reduce costs and improve yield,
• Analyse resourcing models to predict future resource requirements,
• Optimise costs of raw material use,
• Analyse performance of equipment to predict failures,
• Analyse production data to determine what will lead to loss of productivity.
Damian and his team are getting ready to hopefully impress and build a relationship with his
future client. His client has a number of areas they are interested in seeing the powerful tools
of data analytics applied. The details of these are provided in five separate questions. You
should attempt to answer all of them.
2
Question One
Raw materials are used by the prospective client at their manufacturing site, including some
that are sourced from a variety of vendors. Generally, there are no issues with these, apart
from a few exceptions. It has been proposed that Data Science Specialists Ltd. will be provided
with recent data on one particular raw material, which frequently fails to meet the required
specification.
(a) Answer the following:
(i) If provided with a sample of data described as above, the failure rate can be
calculated as follows:
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑜𝑜𝑜𝑜 𝑈𝑈𝑈𝑈𝑈𝑈𝑡𝑡𝑡𝑡 𝑡𝑡ℎ𝑎𝑎𝑎𝑎 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑜𝑜𝑜𝑜 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈
Critically assess whether this value can be used to predict future values.
[3 marks]
(ii) The client has tested ninety batches of this material and data is available for all
of them; however, for expediency, it has been proposed to only provide Data
Science Specialists Ltd. with data from twelve randomly selected batches. You
should critically assess this proposal and state clearly whether or not this is
likely to be sufficient. Justify your answer with three reasons.
[3 marks]
(b) Suppose a dataset on these raw materials was made available containing sixty random
batches. Of these, twenty-four failed to meet the required specifications. In examining the
batches in chronological order, calculate the following probabilities:
(i) One batch selected at random from the sample fails to meet specification.
[2 marks]
(ii) Seven batches selected at random from the sample all fail to meet
specification.
[4 marks]
(iii) Ten batches selected all meet specification.
[4 marks]
Rank these events in terms of how likely they are to occur (starting with the most likely)
[1 mark]
(c) The client is about to make another batch of material. Estimate the likelihood of this batch
passing specification.
[2 marks]
(Total 19 Marks)
3
Question Two
(a) Linear regression analysis is one of the most widely used tools available when analysing
data. A first step in this type of analysis is often calculating a measure of correlation
between variables.
(i) Outline three potential drawbacks in using a linear correlation coefficient
[3 marks]
(ii) Critically assess the appropriateness of establishing causation based on a high
correlation within a dataset such as that listed below?
[3 marks]
(iii) Name one measure of linear correlation and calculate it for the following
dataset. This dataset explores the relationship between the number of
employees working on a given day and the corresponding output in terms of
production units.
Employees (X) 13 16 15 13 14 11 17 17 16 12
Units (Y) 36.4 42.1 40.6 36.2 37.6 28.8 42.3 44.6 41.3 32.7
To aid in your calculations the following are available ∑ 𝑿𝑿𝒊𝒊 = 144 and ∑ 𝒀𝒀𝒊𝒊 = 382.6 and
the standard deviations of X and Y are 2.12 and 4.88 respectively.
[7 marks]
(b) Another dataset the client is interested in analysing compares the independent variable
“raw material expenditure” against the dependent variable “profit”. Using R, a linear
regression analysis was conducted, and the following output generated:
Residuals
Min 1Q Median 3Q Max
-18.7900 -4.6800 .0076 4.6399 17.9402
Coefficients Estimate Std. Error t value Pr(>|t|)
(Intercept) 22.13643 1.61959 13.67 <2e-16 ***
Raw_Materials 2.36677 0.04491 52.70 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 6.77 on 299 degrees of freedom
Multiple R-squared: 0.9028, Adjusted R-squared: 0.9025
F-statistic: 2777 on 1 and 299 DF, p-value: < 2.2e-16
4
(i) Calculate the correlation coefficient from this output and interpret it.
[2 marks]
(ii) Assess the residuals you are seeing? Is there any cause for concern? Give
three reasons for your answer
[3 marks]
(iii) From the coefficients table, can you provide some insight into whether every
term should be included in this model? Explain your answer.
[1 mark]
(iv) To make the decision on what terms should be included in the model in part
(iii), what assumption(s) need to be made about these data?
[1 mark]
(c) The following output is generated from a linear regression analysis comparing the
independent variable “Other Expenditure” against the dependent variable “Profit”.
Coefficients Estimate Std. Error t value Pr(>|t|)
(Intercept) 283.5898 265.5377 1.068 0.286
Other Expenditure -0.1166 0.1733 -0.673 0.502
It has been suggested that this is an appropriate model to be used for prediction.
Critically assess this based on the output above and propose what could be attempted
to improve the model.
[2 marks]
(Total 22 Marks)
5
Question Three
(a) The client’s sales are believed to be seasonal in nature. An analysis has been conducted
by Data Science Specialists Ltd. by breaking these data into the seasonal, trend and
irregular components using a simple additive time series model. These are presented
here:
(i) From this output recommend and justify a suitable methodology to decompose
the time series.
[1 mark]
(ii) With reference to the model you selected in (i), assess what you are likely to
see in the data for the next twelve months. Critically assess how reliable these
predictions will be.
[4 marks]
6
(b) Answer the following:
(i) Critically evaluate the difference between supervised and unsupervised
learning, providing examples of where these techniques could be used.
[7 marks]
(ii) Assess the risks, benefits and challenges of using machine learning.
[6 marks]
(c) Data Science Specialists Ltd. have developed a random forest model to determine which
batches of raw materials will lead to the high or low productivity in production. The model
was developed on a training dataset and subsequently validated on a validation dataset.
On this validation dataset, an accuracy score of 98% was observed.
(i) Explain the term accuracy and how it is calculated in the context of this
question.
[1 mark]
(ii) Give an example of an alternative measure that can be used to assess a
model’s performance in a classification problem.
[1 mark]
(iii) Briefly assess how a random forest works with the aid of an example.
[4 marks]
(Total 24 Marks)
7
Question Four
(a) Data Science Specialists Ltd. have been asked to consider the database which the client
holds on its employees. Consideration is being given to using the current employee’s
personal data in an analysis to determine if manufacturing performance can be improved.
No prior consent has been provided for this by the employees. You should assess what
measures should be taken in advance of undertaking this work to ensure it is done
compliantly.
[3 marks]
(b) Data from clinical trials are being held on the manufacturing site within the European Union
for security purposes. Data Science Specialists Ltd. have been asked what relevant
regulations should be considered in the context of protecting these data particularly in the
area of “the right of access”. Write a short report advising the client of the relevant data
protection regulations and how it applies to the clinical trials, recommending suitable
actions that the client must take.
[15 marks]
(Total 18 Marks)
8
Question Five
(a) Three different suppliers of material are being examined in an exploratory analysis. The
resultant productivity of each supplier’s material is provided below. Critically assess this
output and clearly state the conclusions you draw from this analysis.
Analysis of Variance Table
Df Sum Sq Mean Sq F- Valuee Pr(>F)
Material 2 1.906 0.95295 1.1294 0.326
Residuals 147 124.036 0.84378
[5 marks]
(b) Consider a scenario where three separate models have been applied to a dataset, with
the aim of understanding costs. In terms of comparing the models, list two things that
should be examined and what you would expect to see in each of these.
[6 marks]
Question 5 continues on next page.
9
(c) A principal component analysis was carried out by Data Science Specialists Ltd., to
determine what variables influence yield. The following scores and loadings have been
produced where the numbers represent a unique identifier for each batch (scores) and the
red vectors represent the original variables (which provide a representation of loadings).
Sixty-two percent of the variability in these data has been accounted for by the first
principal component and thirty three percent by the second principal component. Critically
assess this output. (below)
[6 marks]
(Total 17 Marks)
END OF PAPER
10
SUGGESTED SOLUTIONS
THE INSTITUTE OF CERTIFIED PUBLIC ACCOUNTANTS IN IRELAND
DATA ANALYTICS FOR FINANCE
STRATEGIC LEVEL EXAMINATION – SAMPLE PAPER 2021
Question One [19 marks]
(a) Raw materials are used by the prospective client at their manufacturing site, including
some that are sourced from a variety of vendors. Generally, there are no issues with
these, apart from a few exceptions. It has been proposed that Data Science Specialists
Ltd. will be provided with recent data on one particular raw material, which frequently
fails to meet the required specification.
(i) If provided with a sample of data described as above, the failure rate can be
calculated as follows:
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑟𝑟 𝑜𝑜𝑜𝑜 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈 𝑡𝑡ℎ𝑎𝑎𝑎𝑎 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑜𝑜𝑜𝑜 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈
Critically assess whether this value can be used to predict future values.
[3 marks]
[Solution] In this case, the failure rate is calculated based on a sample of
recent data. It is important to note that the recent data are not a random sample
of all historical data and so may not adequately reflect the overall population.
While the calculated rate may be suitable to describe what is being explored in
a short interval, it may not be representative outside of this window. One should
not proceed using this value to predict future values without gaining confidence
that future values will behave similarly to these data. The variability captured
by the data in question may not fully capture the variability in the overall
population due to the sampling method. If this is the case, then using the failure
rate to predict future values will likely lead to erroneous conclusions.
(ii) The client has tested ninety batches of this material and data is available for all
of them; however for expediency, it has been proposed to only provide Data
Science Specialists Ltd. with data from twelve randomly selected batches. You
should critically assess this proposal and state clearly whether or not this is
likely to be sufficient. Justify your answer with three reasons. [3 marks]
[Solution] This is not likely to be sufficient:
• A small sample such as this could lead to hypothesis tests with weak
power.
• With a small sample size, some parts of the population may not be
represented.
• If there is a large standard deviation in the population data, smaller
samples tend to get less representative of the population.
• Using small samples can lead to a bias in conducting inference.
[Note other correct answers will be acceptable here]
11
(b) Suppose a dataset on these raw materials was made available containing sixty
random batches. Of these, twenty-four failed to meet the required
specifications. In examining the batches in chronological order, calculate the
following probabilities:
(i) One batch selected at random from the sample fails to meet specification;
[2 marks]
(ii) Seven batches selected at random from the sample all fail to meet
specification; [4 marks]
(i) Ten batches selected all meet specification. [4 marks]
Rank these events in terms of how likely they are to occur (starting with the most likely)
[1 mark]
[Solution]
Pr(i)=24/60
Pr(ii)=(24/60) (23/59) (22/58) (21/57) (20/56) (19/55) (18/54)=0.000896
Pr(iii)=(36/60) (35/59) (34/58) (33/57) (32/56) (31/55) (30/54) (29/53) (28/52) (27/51)
=0.00337
Order (i), (iii), (ii) based on order of probabilities most likely to least likely.
(c) The client is about to make another batch of material. Estimate the likelihood of this
batch passing specification. [2 marks]
[Solution]
36/60
12
Question Two [22 marks]
(a) Linear regression analysis is one of the most widely used tools available when
analysing data. A first step in this type of analysis is often calculating a measure of
correlation between variables.
(i) Outline three potential drawbacks in using a linear correlation coefficient
[3 marks]
(ii) Critically assess the appropriateness of establishing causation based on a high
correlation within a dataset such as that listed below? [3 marks]
(iii) Name one measure of linear correlation and calculate it for the following
dataset. This dataset explores the relationship between the number of
employees working on a given day and the corresponding output in terms of
production units. [7 marks]
Employees (X) 13 16 15 13 14 11 17 17 16 12
Units (Y) 36.4 42.1 40.6 36.2 37.6 28.8 42.3 44.6 41.3 32.7
To aid in your calculations the following are available ∑ 𝑿𝑿𝒊𝒊 = 144 and ∑ 𝒀𝒀𝒊𝒊 = 382.6 and
the standard deviations of X and Y are 2.12 and 4.88 respectively.
[Solution]
i) Any three of the following or other equivalent answer:
• Where a nonlinear correlation exists, the linear correlation coefficient
may be close to zero in spite of a relationship existing, as it only
examines linear relationships.
• Outliers such as a value recorded incorrectly can have an significant
impact on the calculation
• Where subgroups exist in the dataset but there is no within group
correlation, there can be high values for the linear correlation
coefficient.
• Some correlation coefficients should not be used on an ordinal scale.
ii) It would be inappropriate to establish causation on a dataset such as this.
Spurious correlations do exist and confounding variables can frequently lead
to incorrect conclusions. An obvious example is ice cream sales and flu cases
being correlated. The reason for this is both are linked to the weather. When
the weather gets colder, the flu circulates more in the community and at the
same time the cooler weather makes individuals less likely to buy ice cream.
Accordingly, more stringent measures are needed to imply causation than
correlation in one individual study.
iii) Pearson Product Moment Correlation Coefficient (or similar measure is also
acceptable)
13
X Y X-mX Y-mY (X-mX)*(Y-mY) (X-mX)2 (Y-mY)2
13 36.4 -1.4 -1.86 2.604 1.96 3.4596
16 42.1 1.6 3.84 6.144 2.56 14.7456
15 40.6 0.6 2.34 1.404 0.36 5.4756
13 36.2 -1.4 -2.06 2.884 1.96 4.2436
14 37.6 -0.4 -0.66 0.264 0.16 0.4356
11 28.8 -3.4 -9.46 32.164 11.56 89.4916
17 42.3 2.6 4.04 10.504 6.76 16.3216
17 44.6 2.6 6.34 16.484 6.76 40.1956
16 41.3 1.6 3.04 4.864 2.56 9.2416
12 32.7 -2.4 -5.56 13.344 5.76 30.9136
Total: 90.66 40.4 214.524
𝟗𝟗𝟗𝟗. 𝟔𝟔𝟔𝟔
𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷𝑷 = = 𝟎𝟎. 𝟗𝟗𝟗𝟗𝟗𝟗
√𝟒𝟒𝟒𝟒. 𝟒𝟒 ∗ 𝟐𝟐𝟐𝟐𝟐𝟐. 𝟓𝟓𝟓𝟓𝟓𝟓
14
(b) Another dataset the client is interested in analysing compares the independent variable
“raw material expenditure” against the dependent variable “profit”. Using R, a linear
regression analysis was conducted and the following output generated:
Residuals
Min 1Q Median 3Q Max
-18.7900 -4.6800 .0076 4.6399 17.9402
Coefficients Estimate Std. Error t value Pr(>|t|)
(Intercept) 22.13643 1.61959 13.67 <2e-16 ***
Raw_Materials 2.36677 0.04491 52.70 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 6.77 on 299 degrees of freedom
Multiple R-squared: 0.9028, Adjusted R-squared: 0.9025
F-statistic: 2777 on 1 and 299 DF, p-value: < 2.2e-16
(i) Calculate the correlation coefficient from this output and interpret it.
[2 marks]
[Solution] This is calculated as the square root of the R-Squared value 0.95.
This value being close to unity implies a strong positive linear relationship.
(ii) Assess the residuals you are seeing? Is there any cause for concern? Give three
reasons for your answer [3 marks]
[Solution]
No cause for concern: [Any three of the following are acceptable]
• Data appears to be a random scatter.
• Variance appears constant across the data.
• No patterns visible in the residuals.
• Residuals appear to be centred around zero.
(iii) From the coefficients table, can you provide some insight into whether every term
should be included in this model? Explain your answer. [1 mark]
15
[Solution]
All terms are deemed significant based on the t-test with a significance level of
0.001 and should be included in the model.
(iv) To make the decision on what terms should be included in the model in part (iii),
what assumption(s) need to be made about these data? [1 mark]
[Solution]
It is assumed the errors are independently and identically distributed, following
a normal distribution with a mean of zero and a common standard deviation.
(c) The following output is generated from a linear regression analysis comparing the
independent variable “Other Expenditure” against the dependent variable “Profit”.
Coefficients Estimate Std. Error t value Pr(>|t|)
(Intercept) 283.5898 265.5377 1.068 0.286
Other Expenditure -0.1166 0.1733 -0.673 0.502
It has been suggested that this is an appropriate model to be used for prediction.
Critically assess this based on the output above and propose what could be attempted
to improve the model. [2 marks]
[Solution]
Other expenditure is not considered to be significant in examining the linear
relationship between the predictor and the response variable based on the
hypothesis test at a 5% significance level. It should not be used in the model. Note
this conclusion is only appropriate if the assumption of normality of the errors holds.
Any of the following are acceptable for improving the model [Note this is not an
exhaustive list]
• Using transformations e.g. log or square root .
• Applying weights.
• Adding additional terms to the model.
16
Question Three [24 marks]
(a) The client’s sales are believed to be seasonal in nature. An analysis has been
conducted by Data Science Specialists Ltd. by breaking these data into the seasonal,
trend and irregular components using a simple additive time series model. These are
presented here:
(i) From this output recommend and justify a suitable methodology to
decompose the time series. [1 mark]
[Solution]
The additive model is appropriate here since the seasonal effect does not
change as the trend increases.
(ii) With reference to the model you selected in (i), assess what you are likely to
see in the data for the next twelve months. Critically assess how reliable these
predictions will be. [4 marks]
17
[Solution]
Based on the general increase in the trend of the data over the previous ten
years, it is expected to see this continue; the pattern of seasonality is expected
to remain unchanged and one does not anticipate any change in the random
component into the future.
Note these predictions are based on historic data which may not be a reliable
indicator of future performance. The model is a causal time series model and
so does not consider any predictors other than the data itself. This is potentially
a significant limitation.
(b) Answer the following:
(i) Critically evaluate the difference between supervised and unsupervised
learning, providing examples of where these techniques could be used.
[7 marks]
[Solution]
Supervised learning is using labelled data to build a model for classification to
enable predictions to be made, while in the case of unsupervised learning no
labels/categories are given to the data and it is for the algorithm to draw
inferences without these e.g. cluster analysis.
This means supervised learning involves a mapping function (e.g. regression)
from the predictor variables to the response variable. The aim here is this map
will allow the response values of new predictors be estimated. Unsupervised
learning aims to find the hidden distributions contained within a dataset (e.g.
cluster analysis) and generally does not have response variables it aims to map
to.
Supervised learning can be used in a manufacturing site when exploring yield
of batches. Here a measure of yield is the variable that one aims to predict.
The predictor values could be staff hours used, processing time temperatures
etc. The goal of such analysis could be to determine which parameters can be
varied to increase yield.
Another example of modelling a production process could be where one does
not have a predetermined goal, such as improving yield, but is aiming to gain
insight into patterns in the data. Here tools such as cluster analysis explore
whether there are particular batches behaving in a similar way. This variability
can be explored to gain insights into a manufacturing process and frequently
help point to inconsistencies in raw materials.
(ii) Assess the risks, benefits and challenges of using machine learning.
[6 marks]
[Solution]
Risks/Challenges
18
Based on the fact that many machine learning models are black-box type tools,
an understanding of the mechanics of the modelling is not readily achievable.
As a result of this, bias (systematic over or under estimation) can be an issue
in modelling, leading to inaccurate conclusions. It can be difficult to determine
this until after a model has been applied for a significant period of time.
The size of a dataset is critical when using machine learning models.
Frequently, it is observed that feeding too small a sample size into these modes
results in inaccurate conclusions.
A related issue involves overfitting a model to a training dataset. This can lead
to subsequent poor prediction and erroneous conclusions.
Another issue is that incorrect interpretation of correct model outputs can lead
to poor decision making, particularly when it comes to understanding the
variability of an estimator.
Benefits
Implementation of machine learning models can be quicker than more
traditional statistical models, which require detailed understanding of the
assumptions underlying the models. Additionally, many tools (e.g. the caret
package in R and [Link] in Python, as well as Tensorflow) are available to
support those developing these models. This can make it quite easy to identify
trends and insights from data.
Another benefit of these models is their ability to periodically recalibrate
themselves on the latest data, i.e. to learn and get better. As these models
continuously evolve, their performance is expected to improve over time,
yielding a stronger estimator with potentially less variability
There are many tools available to automate these processes leading to less
resource requirements.
(c) Data Science Specialists Ltd. have developed a random forest model to determine
which batches of raw materials will lead to the high or low productivity in production.
The model was developed on a training dataset and subsequently validated on a
validation dataset. On this validation dataset, an accuracy score of 98% was observed.
(i) Explain the term accuracy and how it is calculated in the context of this
question. [1 mark]
[Solution]
Accuracy is the fraction of data classified correctly
𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁
=
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 + 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁
19
(ii) Give an example of an alternative measure that can be used to assess a
models performance in a classification problem. [1 mark]
[Solution]
Precision is a measure of correct positives expressed in terms of the total
number of positives. In simple terms, this shows what proportion of all positives
were correctly classified.
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 =
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃
(iii) Briefly assess how a random forest works with the aid of an example.
[4 marks]
[Solution]
It can be shown that a large number of uncorrelated trees operating as a
committee will outperform any of the constituent decision trees mathematically.
This is the basis of random forests. The dataset should be sampled with
replacement to generate n training samples. A decision tree should be trained
on each of these n samples. This should be repeated t times. Take the majority
vote if producing categorical values or the average for numeric data.
An obvious example could be where batches are classified as meeting
specification or failing to meet specification based on a number of inputs. A
random forest can be applied to the historic data and used to predict whether
future batches will meet or fail to meet specifications.
20
Question Four [18 marks]
(a) Data Science Specialists Ltd. have been asked to consider the database which the
client holds on its employees. Consideration is being given to using the current
employee’s personal data in an analysis to determine if manufacturing performance
can be improved. No prior consent has been provided for this by the employees. You
should assess what measures should be taken in advance of undertaking this work to
ensure it is done compliantly [3 marks]
[Solution full marks for 3 appropriate answers examples include]
• Ensure informed consent is attained from all those involved before
beginning.
• Attain legal advice from corporate council.
• Ensure systems are compliant with data protection regulations.
(b) Data from clinical trials are being held on the manufacturing site within the European
Union for security purposes. Data Science Specialists Ltd. have been asked what
relevant regulations should be considered in the context of protecting these data
particularly in the area of “the right of access”. Write a short report advising the client
of the relevant data protection regulations and how it applies to the clinical trials,
recommending suitable actions that the client must take. [15 marks]
[Solution full marks for any three correct points]
As the company is within the European Union (EU), the relevant document is
the general data protection regulations sections related to the right of access.
The subjects in the clinical trial are entitled to access a number of items.
All subjects the company holds data on are entitled to know how the data about
them was acquired and the purposes of the processing of their data, including
if machine learning type algorithms are applied to their data for profiling.
Generally the reasons for using the data should be made available to the
participants on request. This is key and can be quite easy for a company to
unwittingly fall foul of. In a scenario where a member of staff conducts an ad
hoc analysis on a dataset such as this, they may conduct work the subject has
not agreed to. This may be a breach of the regulation. Similarly, if work done is
not documented, the company may not be able to provide a record of all
analyses done on the individuals data upon request.
There are different types of data available when conducting a clinical trial and
this could include attributes such as name, address, date of birth as well as
critical medical information. The categories of data being maintained must be
made available to the subject on request. While this may be quite simple in
terms of querying the data from a database, it is important that the structure of
the databases are such that some of these data are not overlooked.
If a clinical trial is successful, then the participants data will be shared with a
number of regulators, analysis of the data may be outsourced to a third party,
21
clinicians may wish to review the data at the end of the study and the company
may wish to present the results to investors. All participants have a right to
know the recipients or categories of recipient to whom the personal data have
been or will be disclosed, in particular recipients in third countries (non EU
member) or international organisations.
It is not always clear how long data will need to be stored for and in the case of
clinical trials, there may need to be larger trials conducted before submitting
data to a regulator or other delays. In this case, it may not be possible to share
the length of time that the data will be stored but the criteria which will lead to
its deletion will need to be made available on request.
The subject also has the right to request their data be erased. In this case, if a
subject did not complete the trial and wishes to have their data deleted, this will
have to be done unless there is another legal mechanism in place to deal with
this. The subject at all times retains the right to request their data be deleted
and to lodge a complaint with a supervisory authority. There are exceptions to
this and in the event of a request to delete their data, this should be discussed
by the data controller and corporate council.
22
Question Five [17 marks]
(a) Three different suppliers of material are being examined in an exploratory analysis.
The resultant productivity of each supplier’s material is provided below. Critically
assess this output and clearly state the conclusions you draw from this analysis.
[5 marks]
Analysis of Variance Table
Df Sum Sq Mean Sq F- Valuee Pr(>F)
Material 2 1.906 0.95295 1.1294 0.326
Residuals 147 124.036 0.84378
[Solution]
The medians appear similar in the boxplots, however there appears to be less varia
bility present for the third material. ANOVA tests the null hypothesis that the means
for all groups are the same. In this case, a p-value of 0.326 has been found, which
does not provide significant evidence to reject the null hypothesis at a 5% significan
ce level. What appears missing from this output is a hypothesis test to assess if the
variability across groups is different. In this case, all that can be concluded is there i
s no significant difference in terms of the means of the groups, but it would be worth
considering the variability. There is also no output concerning checking the assu
mptions of the model. Assumptions such as normality of the residuals cannot be ve
rified here.
(b) Consider a scenario where three separate models have been applied to a dataset, with
the aim of understanding costs. In terms of comparing the models, list two things that
should be examined and what you would expect to see in each of these.
[6 marks]
[Solution]
• A plot of residuals should be viewed. In this case, there should be no pattern
in the residuals, they should be unbiased and have a constant variance
independent of any x or y variable.
23
• Plot of the model fitted to data. In this case, the model should fit the data
equally well for any values of x and y, without significant outliers or leverage
points.
• Other items which could be examined include model comparison e.g. F-
test and assumptions of models, measures of goodness of fit etc.
c) A principal component analysis was carried out by Data Science Specialists Ltd.,
to determine what variables influence yield. The following scores and loadings
have been produced where the numbers represent a unique identifier for each
batch (scores) and the red vectors represent the original variables (which provide
a representation of loadings). Sixty-two percent of the variability in these data has
been accounted for by the first principal component and thirty three percent by the
second principal component. Critically assess this output.
[6 marks]
[Solution]
95% of the variability in the data is accounted for by the first two principal
components. This means it is unlikely that the analysis of subsequent components
will yield significant value, as only 5% of the variability in these data remain to be
explained by subsequent components. The first principal component appears to be
dominated by the capital expenditure variable, which makes very little contribution to
the second principal component. The largest contributor to the second component is
processing time, with very little contribution from material cost and yield.
24
It is worth examining if material cost and yield appear to dominate the third and fourth
principal components, as if this is the case, the dimension reduction is not likely to be
successful. It is not entirely clear from this image if there is a high degree of
correlation between variables that can be exploited for analysis.
25