REVIEW OF BASIC DATA ANALYTIC METHODS USING R

The previous chapter presented the six phases of the Data Analytics Lifecycle.

• Phase 1: Discovery

• Phase 2: Data Preparation

• Phase 3: Model Planning

• Phase 4: Model Building

• Phase 5: Communicate Results

• Phase 6: Operationalize

The first three phases involve various aspects of data exploration. In general, the success of a da ta
analysis project requires a deep understanding of the data. It also requires a toolbox for mining and pre-
senting the data. These activities include the study of the data in terms of basic statistical measures and
creation of graphs and plots to visualize and identify relationships and patterns. Several free or commercial
tools are available for exploring, conditioning, modeling, and presenting data. Because of its popularity and
versatility, the open-source programming language Ris used to illustrate many of the presented analytical
tasks and models in this book.
This chapter introduces the basic functionality of the Rprogramming language and environment. The
first section gives an overview of how to useR to acquire, parse, and filter the data as well as how to obtain
some basic descriptive statistics on a dataset. The second section examines using Rto perform exploratory
data analysis tasks using visua lization. The final section focuses on statistical inference, such as hypothesis
testing and analysis of variance in R.

3.1 Introduction toR

Ris a programming language and software framework for statistical analysis and graphics. Available for use
under the GNU General Public License [1], Rsoftware and installation instructions can be obtained via the
Comprehensive RArchive and Network [2]. This section provides an overview of the basic functionality of R.
In later chapters, this foundation in Ris utilized to demonstrate many of the presented analytical techniques.
Before delving into specific operations and functions of Rlater in this chapter, it is important to under-
stand the now of a basic Rscript to address an analytical problem. The following Rcode illustrates a typical
analytical situation in which a dataset is imported, the contents of the dataset are examined, and some
modeling building tasks are executed. Although the reader may not yet be familiar with the R syntax,
the code can be followed by reading the embedded comments, denoted by #. In the following scenario,
the annual sales in U.S. dollars for 10,000 retail customers have been provided in the form of a comma-
separated-value (CSV) file. The read . csv () function is used to import the CSV file. This dataset is stored
to the Rvariable sales using the assignment operator <- .

H imp< rt a CSV file of theo tot'l.- annual sa es for each customcl·

sales <- read . csv( "c:/data/yearly_sales.csv")

# €'>:amine Lhe imported dataset

head(sales)
 3.1 Introduction toR

summary (sales)

# plot num_of_orders vs. sales

plot(sales$num_of_orders,sales$sales_total,
main.. "Number of Orders vs. Sales")

# perform a statistical analysis (fit a linear regression model)

results <- lm(sales$sales_total - sales$num_of_orders)
summary(results)

# perform some diagnostics on the fitted model

# plot histogram of the residuals
hist(results$residuals, breaks .. 800)

In this example, the data file is imported using the read. csv () function. Once the file has been
imported, it is useful to examine the contents to ensure thatthe data was loaded properly as well as to become
familiar with the data. In the example, the head ( ) function, by default, displays the first six records of sales.

# examine the imported dataset

head(sales)
cust - id sales - total num - of - orders gender
100001 800.64
2 100002 217.53

4
100003
100004
5 100005
6 100006
TRACE KTU
74.58
·198. 60
723.11
69.43
2

4
2
t·l
t•l
F
F

The summary () function provides some descriptive statistics, such as the mean and median, for
each data column. Additionally, the minimum and maximum values as well as the 1st and 3rd quartiles are
provided. Because the gender column contains two possible characters, an "F" (female) or "M" (male),
the summary () function provides the count of each character's occurrence.
summary(sales)

cust id sales total num of orde1·s gender

i"lin. :100001 !'-lin. 30.02 !'-lin. 1.000 F:5035
1st Qu . : 1 o25o 1 1 s t Qu . : 80 . 29 1st Qu.: 2.000 !·l: 4965
l>ledian :105001 r'ledian : 151.65 t>ledian : 2.000
!>lean :105001 t•lean 24 9. 46 !>lean 2.428
3rd Qu. :107500 3 rd Qu. : 2 9 5 . 50 3rd Qu.: 3.000
l\lax. : 110 000 t•lax. :7606.09 1•1ax. :22.000

Plotting a dataset's contents can provide information about the relationships between the vari-
ous columns. In this example, the plot () function generates a scatterplot of the number of orders
(sales$num_of_orders) againsttheannual sales (sales$sales_total). The$ is used to refer-
ence a specific column in the dataset sales. The resulting plot is shown in Figure 3-1.
# plot num_of_orders vs. sales
plot(sales$num_of_orders,sales$sales_total,
main.. "Number of Orders vs. Sales")
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

Number of Orders vs. Total Sales

0
iii 0
0 0 0
:§ 0
I <0
C/)
Q>
0 0
iii
C/)
§ 0 0 0
0
II>

i
0
8 ~
I I I I i•
8
i 0
C/)
Q>
0 0 0 0
0
iii
C/)
N
8
0
• 5 10 15 20

sales$num_of_orders

FtGURE 3-1 Graphically examining the data

Each point corresponds to the number of orders and the total sales for each customer. The plot indicates
that the annual sales are proportional to the number of orders placed. Although the observed relationship
between these two variables is not purely linear, the analyst decided to apply linear regression using the
lm () function as a first step in the modeling process.

r esul t s <- lm(sal es$sa l es_total - sales$num_of _or de rs)

r e s ults

ca.l:
lm formu.a

Coefti 1en·
TRACE KTU sa.c Ssales ~ota. sales$num_of_orders

In· er ep • sa:essnum o f orders

The resulting intercept and slope values are -154.1 and 166.2, respectively, for the fitted linear equation.
However, results stores considerably more information that can be examined with the summary ()
function. Detailson thecontents of results are examined by applying the at t ributes () function.
Because regression analysis is presented in more detail later in the book, the reader should not overly focus
on interpreting the following output.

summary(results)

Call :
lm formu:a sa!esSsales_total - salcs$ num_of_orders

Re!'a ilnls:
Min IQ Med1an 3C 1·1ax
-666 . 5 12S . S - 26 . 7 86 . 6 4103 . 4

Coe f ficie nt:s:

Est1mate Std . Errol r value Prl> t
Intercept -15~.128 .; . 12"' - 37 . 33 <2e-16
sal~s$num f orders 166 . 22l 1 . 462 112 . 66 <2e - ~6
 3.1 Introduction to R

Sior:1t . codes : 0 ' ... . 0.00! •·· · c . o: •• • 5 I • I .1 I 1

Res1aua. star:da~d e~ro~ : ~! .: on 999° deg~ees of :reeo~m

~ultlple R·squar d : 0 . ~617 , Aa:usted P-sq~a~ed : . 561

The summary () function is an example of ageneric function. A generic function is a group of fu nc-
tions sharing thesame name but behaving differently depending on the number and the type of arguments
they receive. Utilized previously, plot () is another example of ageneric function; the plot is determined
by the passed variables. Generic functions are used throughout this chapter and the book. In the final
portion of the example, the following Rcode uses the generic function hist () to generate a histogram
(Figure 3-2) of the residualsstored in results. The function ca ll illustrates that optional parameter values
can be passed. In this case, the number of breaks is specified to observe the large residuals.

~ pert H. some d13gnosLics or. the htted m..de.

# plot hist >gnm f the residu, ls
his t (r esults $res idua l s, breaks= 8 00)

Histogra m of resultsSresid uals

TRACE KTU
0
I()

u>-
..
c
:J
<:T
0
0

~ 0
u. I()

0 1000 2000 3000 4000

resuttsSres1duals
FIGURE 3-2 Evidence oflarge residuals

This simple example illustrates a few of the basic model planning and building tasks that may occur
in Phases 3 and 4 of the Data Analytics Lifecycle. Throughout this chapter, it is useful to envision how the
presented Rfunctionality will be used in a more comprehensive analysis.

3.1.1 R Graphical User Interfaces

Rsoftware uses a command-line interface (CLI) that is similar to the BASH shell in Li nux or the interactive
versionsof scripting languages such as Python. UNIX and Linux userscan enter command Rat the terminal
prompt to use the CU. For Windowsinstallations, Rcomes with [Link], which provides a basic graphica l
user interface (GUI). However, to im prove the ease of writing, executing, and debugging Rcode, several
additional GUishave been written for R. Popular GUis include the Rcommander [3]. Rattle [4], and RStudio
[5). Thissection presentsa brief overview of RStudio, which was used to build the Rexamples in th isbook.
Figure 3-3 provides a screenshot of the previous Rcode example executed in RStudio.
REVIEW O F BASIC DATA ANALYTIC M ETHODS USING R

._CNtl.
.. - - ....._..
.. tJ
-....y
- ..... O....ft• f

....".,
' 1 • -...1 • n •• "'
.:1-
t 1 ules r t -.[Link] 6llu,-...,.1.,_uh,1.u.,· .-------, ulu 10000 OM. ol " " whbhs
Scripts
...
...
tw..o u ln
!~uln •
rn~o~ hs
j Workspace
...
.......•. . '
.........
plot

ruulu
u1ts~of-orcl9ors,u1es

t"ltSIIIU
to' 1 '
1• uln ,u lu_utul
Jo
ules_tot•l,
flt I
u t n- ·~ ~

ulu ~ ,..._ot_orct«'s
T
-
cwo...s '"· s..ln

.... ,..,.,...
; :- •toMtt·
""'-
0 { a......
•. ' 11 luu
'"
lU •Ill rt f " ~ I
Ul hht ruuhs 1 t"tst~•h. br u11 • 100
r Histogram or results$reslduals

••'"• iu~-~.~...~------:=::::::::::::::::~
"s-uy(resulu)
·- .:1
~ Plots
c•H:
l•(f or-..1& - ulu lulu_uul .. saln~of_orcMf's)

... , .... ls :
tUn
· 6M. 1 - US, \
tQ lllt'dhn
•U .7
1Q ~b
t6.6 .&10), <1
Console ~ ~

lsttuu Ud. Crrw 1 " ' ',.,. k(,.Jtl)

j
omuupt) .,,,,u, •.ut -Jr.n ~ •.,, •·•
u l•s~oLoro.-' u6. Ul 1. 41 61 uJ.M <-lt- 16 ••• ~
sttnU . codts : o '• •• ' o. oot •••• 0.01 ••• [Link] •, • 0.1 • • 1
l:utdlu•1 sund¥d uror: 110.1 on tnt CS•vr"' or fr~
tty\ttph I:•S.,¥-.1: 0 , $617, .lodJw~ t f'd l•[Link]'d: 0 . )6)7

j
f" • Uathttc : t . ltl••Ool on I wrd 9Ht Of", P•VA1U41: ot l . h•16

,. #pf'rfor• ~- diA~ltn Of' Uw ttlud _,.., 1000 2000 3000

Jo ?lot hhtOQrM ot tt,. re~lduh
~ Mn(ruwh~k'n l cN .th, .,.,,,, · tOO)

FIGURE 3-3 RStudio GUI

TRACE KTU
The fou r highlighted window panes follow.

• Scripts: Serves as an area to writeand saveR code

• Workspace: Lists the datasets and variablesin the Renvironment

• Plot s: Displays the plots generated by the Rcode and provides astraightforward mechanism to
export the plots

• Console: Provides a history of theexecuted Rcode and the output

Additionally, the console pane can be used to obtain help information on R. Figure 3-4 illustrates that
by entering ? lm at the console prompt, the help details of the lm ( ) function are provided on the right.
Alternatively, help (lm ) could have been entered at the console prompt.
Functions such as edit () and fix () allow the user to update the contents of an Rvariable.
Alternatively, such changes can be implemented with RStudio by selecting the appropriate variable from
the workspace pan e.
Rallows one to save the workspace environment, includ ing variables and loaded libraries, into an
. Rdata file using the save . image () function. An existing . Rdata file can be loaded using the
load . image () function. Tools such as RStudio prompt the user for whether the developer wants to
save the workspace connects prior to exiting the GUI.
The reader is encouraged to install Rand a preferred GUI to try out the Rexamples provided in the book
and utilize the help functionality to access more details about the discussed topics.
 3.1 Introduction toR

.._. . -;:,
...,
t" u lu
t
r e...[Link]
tt • • ...........,
.,.1yJ•1n.u\
41 1ft f Ot l.a.l:?' ~~ .:J-
J U
ulu
.. ...... o.ttl01 • i ~·

',
..
411\1 , •

......" ..
N
ru~o~lu

..,"'..,
hud ulu
s~salu
,. ,. ..
plot ultJ l......_.,....ot'MrS ,[Link]. .. ,,.. -~of arden'''~· ~·u·

... t>t .II It~ r--r;~l~ ~\)

,,..
>01 • f • .I H-It t II
[Link] 1• uluSulu,.ICKil ulu s~of ..crd~
>07 ru~l n
J ,.. .... ,~ """
110 "' . . . .z..
Ul "r•·f dl \1 nt,,d~l ll f"'*"t 1--Ut M-" '"
U7 •ph•t hi t< t t~ ' " ' ••I
lU hht ru11hs Srut duah, bru~s • 100 1
u•
~••
,;.·,;·iah.... ; ;.;: .:---------=========----' ---
Fitting Linear Models

un:
.:J o..c:ttptSon
lo(f [Link] .. ultsSul•s..tO'CAI .. uo hsJtuil,..of_or~s )
~-u&t411 kllftl•f"'Idek I CM ... vt. . IIC:WfYWft9'*1 - UIQ!It1UIIIIIII~II-I .nd
[Link] : 1Nfyt4vlt- • (~ • · ~~ . _.,_...clfbab'UMn)
Min IQ ..fltiM )Q '""'
~tM. , · US. S • lt.7 M , t 4110), 41

CCHff lc.l..r:ts: uqrcrw:a!• , 41 c..•, , ....., , , , .,.l111ht•, ft• .• [Link],

(fnurcept)
Uttaau St d. lrror t "'' ' "' "'(•I t I>
-1~ .UI <[Link] - [Link] c2t-l6 ••·
_,bOd • • q" · · .o., .. T~. • .. ~. y .. r;u..,r, ql" • ru:z.
• ~ h :.c t • TWI, cMtr uu • II"J:.I., o thu, . .. )
u 1e~ lnwa..ot _orws 1641.211 1."62 UI.M -.l t - 16 ...
Argument1
stvnu. cCIIOH.: o ···-· 0.001 •••• 0.01 ••• o.o\ ·.• o. t • • 1
• ntdvll nln<t¥0 tf'ror: 210.1 on t9N detJ't.S of frt~ f QI&l}& lf'lot.,Kt~tl&•••t· t• liii.•(OfiDMII'IIfunbece«cldlotNIWIII t tymtdcestiiCI¥IbOnol I

::::::.~.:.~:-:=',::7'-=:::.7::~·~ 101CIU
tt~~lt
tple •-•qwwtd: 0. MJ7. AdJW1ttO •·squAred: o. ~617
r -su thttc : l.2t2...o.& on 1 1t10 tttl cw , .,._., ,..,.: c 2.21-16 I
......)t. . . . . . tlw""'*''"tlltfNdee fnalbn:[Link]&:.• tt.~. . . tll... hm
) • .,..,.,0,.. · - ctl~tt( \ 01'1

...,.
• • plot ,.,,uq•
of ttw rnldv• h
.. "'ht(r~whs ~ ~sl..,.h, lilr'Uh • toO)

-• I
tr.. flt\f'G . . , . ,

-- ,-
J..... u ... tr_::~, f t ;....,l l l tYJIIUI1CIIt.,....,....,.kwaiiiii'O~•uhcl
II'I~'IIIKtorl~ l ....... fl .......... lO beuMO ~UMtc,.,poctst
II'I__......[Link]....,_,teMI<MCI•t!lotiftaltJI'X-tsl SPIOII4 c.r--..:.or•-...aor
1--'UL ....,C~IM1Il ...... d•l4~..ql l vt:~t•~•
, ..r..... -,,, ...,.,...,.,.,.,,.,... ...,., .. YMO S..[Link]
~ng
.=.J

FIGURE 3-4 Accessing help in Rstudio

3.1.2 Data Import and Export

TRACE KTU
In the annual retail sales example, the dataset was imported into R using the read . csv () function as
in the following code.

sales <- read . csv("c : /data/yearly_ sales . csv" )

R uses a forward slash {!) as the separator character in the directory and file paths. This convention
makes script Iiies somewhat more portableat the expense of some initial confusion on the part of Windows
users, w ho may be accustomed to using a backslash (\) asa separator. To simplify the import of multiple Iiies
with long path names, the setwd () function can be used to set the working directory for the su bsequent
import and export operations, as shown in the follow ing R code.

setwd ("c: / data/ ")

sales < - [Link] ( "yearly_sales . csv" )

Other import functions include read. table ( l and read . de lim () ,which are intended to import
other common file types such as TXT. These functions can also be used to import the yearly_ sales
. csv file, as the following code illustrates.

sales_table <- read .table ( "yearly_sales . csv " , header=TRUE, sep="," )

sales_delim <- read . delim ("yearly_sales . csv", sep=",")

Th e ma in difference between these import functions is the default values. For example, t he read
. delim () function expectsthe column separator to be a tab("\ t"). ln the event that the numerical data
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

in a data file uses a comma for the decimal, Ralso provides two additional functions-read . csv2 () and
read . del im2 ()-to import such data. Table 3-1includes the expected defaults for headers, column
separators, and decimal point notations.

TABLE 3-1 Import Function Defaults

Function Headers Separator Decimal Point

r ead. t abl e () FALSE
r ead. csv () TRUE
r ead. csv2 ( ) TRUE .
"·"

read .d elim () TRUE "\t "

r ead. d elirn2 () TRUE "\t " ...

The analogous Rfunctions such as write . table () ,write . csv () ,and write . csv 2 () enable
exporting of Rdatasets to an external fi le. For example, the following Rcode adds an additional column
to the sales dataset and exports the modified dataset to an external file.

t; ?\dd 1 ,... ... l,;.f'1Il t .. • h.;;. _'b:!l JUt :i ~ ~ j

sales$per_order

# exp 1L TRACE KTU

d1ta 1s
<- sa l es$sales_total/sales$num_of _orders

''ll' 1.-.. <r u 1 "' L!1< llL

write . t able(sales ," sa l es_modified .txt ", sep= "\t ", row. names=FALSE
t n• t '1'.' name,,

Sometimes it is necessary to read data from a database management system (DBMS). Rpackages such
as DBI [6) and RODBC [7] are available for this purpose. These packages provide database interfaces
for communication between Rand DBMSs such as MySQL, Oracle, SQL Server, PostgreSQL, and Pivotal
Greenplum. The following Rcode demonstrates how to instal l the RODBC package with the i ns t al l
. p acka ges () function. The 1 ibr a ry () function loads the package into the Rworkspace. Finally, a
connector (conn ) is initialized for connecting to a Pivotal Greenpl um database tra i n i ng2 via open
database connectivity (ODBC) with user user. The training2 database must be defined either in the
I etc/ODBC . ini configuration file or using the Administrative Tools under the Windows Control Panel.
install . packages ( "RODBC" )
library(RODBC)
conn <- odbcConnect ("t r aining2", uid="user" , pwd= "passwor d " )

Th e con nector needs to be present to su bmit a SQL query to an ODBC database by using the
sq l Qu ery () function from the RODBC package. The following Rcode retrieves specific columns from
the housi ng table in which household income (h inc ) is greater than $1,000,000.

housing_data s qlQuery(conn, "select s erialno , s t ate, persons, r ooms

<-
from housing
where hinc > 1000000")
head(housing_data )
 3.1 Introduction t o R

4552088 5 9
4 45"- 88 5 9
5 8699:!93 6 5 5

Although plots can be saved using the RStudio GUI, plots can also be saved using Rcode by specifying
the appropriate graphic devices. Using the j peg () function, the following Rcode creates a new JPEG
file, adds a histogram plot to the file, and then closes the file. Such techniques are useful w hen automating
standard repor ts. Other functions, such as png () , bmp () , pdf () ,and postscript () ,are available
in Rto save plots in the des ired format.

jpeg ( fil e= "c : /data/ sale s_h ist . j peg" ) creaLe a ne'" jpeg file
h ist(sales$num_of_ o rders ) # export histogt·;un to jpeg
d ev. o ff () ~ shut off the graphic device

More information on data imports and exports can be fou nd at http : I I cran . r-proj e ct . o rgl
doc I ma nuals I r- rel ease i R- da ta . html, such as how to import datasets from statistical software
packages including Minitab, SAS, and SPSS.

3.1.3 Attribute and Data Types

In the earlier exa mple, the sal es variable contained a record for each customer. Several cha racteristics,
such as total an nual sa les, number of orders, and gender, were provided for each customer. In general,

TRACE KTU
these characteristics or attributes provide the qualitative and quantitative measures for each item or subject
of interest. Attributes can be categorized into four types: nominal, ordinal, interval, and ratio (NOIR) [8).
Table 3-2 distinguishes these four attrib ute types and shows the operations they support. Nominal and
ordinal attributes are considered categorical attributes, w hereas interval and ratio attributes are considered
numeric attributes.

TABLE 3-2 NOIR Attribute Types

Categorical (Qualitative) Numeric (Quantitative)

Nominal Ordinal Interval Rat io

Definition The va lues represent Attributes The difference Both the difference
labels that distin- imply a betw een two and the ratio of
guish one from sequence. values is two values are
another. meaningful. meaningful.

Examples ZIP codes, nationa l- Quality of Temperature in Age, temperature

ity, street names, diamonds, Celsiusor in Kelvin, counts,
gender, employee ID academic Fahrenheit, ca l- length, weight
numbers, TRUE or grades, mag- endar dates,
FALSE nitude of latitudes
ea rthquakes

Operations =, >' =, ~, = , ;t., =,~,

<, s , >, 2: <, s , >, c:, <, s , >, ~,

+, - +, - ,

x, .:-
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

Data of one attribute type may be converted to another. For example, the qual it yof diamonds {Fair,
Good, Very Good, Premium, Ideal} is considered ordinal but can be converted to nominal {Good, Excellent}
with adefined mapping. Similarly, aratio attribute like Age can be converted into an ordinal attribute such
as {Infant, Adolescent, Adult, Senior}. Understanding the attribute types in a given dataset is important
to ensure that the appropriate descriptive statistics and analytic methods are applied and properly inter-
preted. For example, the mean and standard deviation of U.S. postal ZIP codes are not very meaningful or
appropriate. Proper handling of categorical variables will be addressed in subsequent chapters. Also, it is
useful to consider these attribute types during the following discussion on Rdata types.

Numeric, Character, and Logical Data Types

Like other programming languages, Rsupports the use of numeric, character, and logical (Boolean) values.
Examples of such variables are given in the following Rcode.
i <- 1 # create a numeric variable
sport <- "football" # create a character variable
flag <- TRUE # create a logical variable

Rprovides several functions, such as class () and typeof (),to examine the characteristics of a
given variable. The class () function represents the abstract class of an object. The typeof () func-
tion determines the way an object is stored in memory. Although i appears to be an integer, i is internally
stored using double precision. To improve the readability of the code segments in this section, the inline

class(i)
typeof(i)
TRACE KTU
Rcomments are used to explain the code or to provide the returned values.

# returns "numeric"
# returns "double"

class(sport) # returns "character"

typeof(sport) # returns "character"

class(flag) ..
ttreturns "logical"
typeof (flag) # returns "logical"

Additional Rfunctions exist that can test the variables and coerce a variable into a specific type. The
following Rcode illustrates how to test if i is an integer using the is . integer ( } function and to coerce
i into a new integer variable, j, using the as. integer () function. Similar functions can be applied
for double, character, and logical types.
[Link](i) # returns FALSE
j <- [Link](i) # coerces contents of i into an integer
[Link](j) # returns TRUE

The application of the length () function reveals that the created variables each have alength of 1.
One might have expected the returned length of sport to have been 8 for each of the characters in the
string 11 football". However, these three variables are actually one element, vectors.
length{i) # returns 1
length(flag) # returns 1
length(sport) # returns 1 (not 8 for "football")
 3.1 Introduction to R

Vectors
Vectors are abasic building block for data in R. As seen previously, simple Rvariables are actually vectors.
A vector can only consist of values in the same class. The tests for vectors can be conducted using the
is. vector () function.
[Link](i) !t returns TRUE
[Link](flag) # returns TRUE
[Link](sport) ±t returns TRUE

Rprovides functionality that enables the easy creation and manipulation of vectors. The following R
code illustrates how a vector can be created using the combine function, c () or the colon operator, :,
to build a vector from the sequence of integers from 1 to 5. Furthermore, the code shows how the values
of an existing vector can be easily modified or accessed. The code, related to the z vector, indicates how
logical comparisons can be built to extract certain elements of a given vector.
u <- c("red", "yellow", "blue") " create a vector "red" "yello•d" "blue"
u ±; t·eturns "red" "yellow'' "blue"
u[l] returns "red" 1st element in u)
v <- 1:5 # create a vector 1 2 3 4 5
v # returns 1 2 3 4 5
sum(v) It returns 15
w <- v * 2 It create a vector 2 4 6 8 10
w
w[3]
z
z
z
<-

> 8
v + w TRACE KTU # returns 2 4 6 8 10
returns 6 (the 3rd element of w)
# sums two vectors element by element
# returns 6 9 12 15
# returns FALSE FALSE TRUE TRUE TRUE
z [z > 8] # returns 9 12 15
z[z > 8 I z < 5] returns 9 12 15 ("!"denotes "or")

Sometimes it is necessary to initialize a vector of a specific length and then populate the content of
the vector later. The vector ( } function, by default, creates a logical vector. Avector of a different type
can be specified by using the mode parameter. The vector c, an integer vector of length 0, may be useful
when the number of elements is not initially known and the new elements will later be added to the end
ofthe vector as the values become available.
a <- vector(length=3) # create a logical vector of length 3
a # returns FALSE FALSE FALSE
b <- vector(mode::"numeric 11 , 3) #create a numeric vector of length 3
typeof(b) # returns "double"
b[2] <- 3.1 #assign 3.1 to the 2nd element
b # returns 0.0 3.1 0.0
c <- vector(mode= 11 integer", 0) # create an integer vectot· of length o
c # returns integer(O)
length(c) # returns o

Although vectors may appear to be analogous to arrays of one dimension, they are technically dimen-
sionless, as seen in the following Rcode. The concept of arrays and matrices is addressed in the following
discussion.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

length(b) 1·eturns 3
dim(b) ~ 1·etun1s NULL (an undefined value)

Arrays and Matrices

The array () function can be used to restructure avector as an array. For example, the following Rcode
builds a three-dimensional array to hold the quarterly sales for three regions over a two-year period and
then assign the sales amount of $158,000 to the second region for the first quarter of the first year.

H the dimensions are 3 regions , 4 quarters, and 2 years

quarterly_sales < - array(O, dim=c(3,4,2))
quarterly_sales[2,1,11 <- 158000
quarterly_sales

[. 1, [.:! 1 :. , 1 [,·s!
[1' 1 0 0 0
[:!,1 !58000 c 0 0
[ 3. 1 0 0 0 0

[1' 1
[2 ' 1
TRACE KTU
[. 11 (.21 [. 31 [. 41
0
0
0
0
0
0
0
0
[3 ' 1 0 0 0 0

A two-dimensional array is known as a matrix. Thefollowing code initializes a matrix to hold the quar-
terly sales for thethree regions. The parameters nrov1 and nco l define the number of rows and columns,
respectively, for the sal es_ma tri x.

sales_matrix <- matrix(O, nrow = 3, neal 4)

sales_matrix

[.11 ;,:!) 1.31 [. .; 1

[ 1, 1 0 0 0 0
[2,1 0 0 0 0
[ 1' 1 n

Rprovides the standard matrix operations such as addition, subtraction, and multiplication, as well
as the transpose function t () and the inverse matrix function ma t r ix . inve r s e () included in the
matrixcalc package. Th e following Rcode builds a 3 x 3 matrix, M, and multiplies it by its inverse to
obtain the identity matrix.

library(matrixcalc)
M <- matrix(c(1,3,3,5,0,4,3 , 3,3) ,nrow 3,ncol 3) build a 3x3 matrix
 3.1 Introduction toR

M %* % matrix . inverse (M} ~ multiply 1·! by inverse (:01}

[. 1] [. 2] [ ' 3]
[1' J 0 0
[2' J 0 1 0
[3' J 0 0 1

Data Frames
Similar to the concept of matrices, data frames provide astructure for storing and accessing several variables
of possibly different data types. In fact, asthe i s . d ata . fr a me () function indicates, a data frame was
created by the r e ad . csv () function at the beginning of the chapter.

[Link] a CSV :ile of the total annual sales :or each customer
s ales < - read . csv ("c : / data/ ye arly_s a l es . c sv" )
i s .da t a . f r ame (sal es ) ~ t·eturns TRUE

As seen earlier, the variables stored in the data frame can be easily accessed using the $ [Link]
following Rcode illustrates that in this example, each variable is a vector with the exception of gende r ,
which was, by a read . csv () default, imported as a factor. Discussed in detail later in thissection, a factor
denotes a categorical variable, typically with a few finite levels such as "F" and "M " in the case of gender.

l ength(sal es$num_of _or ders) returns 10000 (number of customers)

TRACE KTU
i s . v ector(sales$cust id)
-
is . v ector(sales$sales_total)
i s .vector(sales$num_of_orders )
returns
returns
returns
TRUE
TRUE
TRUE
is . v ector (sales$gender) returns FALSE

is . factor(s a les$gender ) ~ returns TRUE

Because of their flexibility to handle many data types, data frames are the preferred input format for
many ofthe modeling functions available in R. The foll owing use of the s t r () function provides the
structure of the sal es data frame. This function identifi es the integer and numeric (double) data types,
the factor variables and levels, as well as the first few values for each variable.

str (sal es) # display structure of the data frame object

'[Link]·ame': 10000 obs . of 4 vanables :

$ CUSt id int 100001 100002 100003 100004 100005 100006 ...
$ sales total num 800 . 6 217.5 74.6 498 . 6 723 . 1
$ num of - orders : int 3 3 2 3 4 2 2 2 2 2 . ..
$ gender Factor w/ 2 le,·els UfU "f'-1" : 1 l 2 2 1 1 2 2 1 2 .. .
I

In the simplest sense, data frames are lists of variables of the same length. A subset of the data frame
can be retrieved through subsetting operators. R's subsetting operators are powerful in t hat they allow
one to express complex operations in a succinct fashion and easily retrieve a subset of the dataset.

'! extract the fourth column of the sales data frame

sal es [, 4]
H extract the gender column of the sales data frame
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

sales$gender
# retrieve the first two rows of the data frame
sales[l:2,]
# retrieve the first, third, and fourth columns
sales[,c(l,3,4)]
l! retrieve both the cust_id and the sales_total columns
sales[,c("cust_id", "sales_total")]
# retrieve all the records whose gender is female
sales[sales$gender=="F",]

The following Rcode shows that the class of the sales variable is a data frame. However, the type of
the sales variable is alist. A list is acollection of objects that can be of various types, including other lists.
class(sales)
"data. frame"
typeof(sales)
"list"

Lists
Lists can contain any type of objects, including other lists. Using the vector v and the matrix M created in
earlier examples, the following Rcode creates assortment, a list of different object types.

TRACE KTU
# build an assorted list of a string, a numeric, a list, a vector,
# and a matrix
housing<- list("own", "rent")
assortment <- list("football", 7.5, housing, v, M)
assortment

[ [1)]
[1) "football"

[ (2])
[1) 7. 5

[ (3])
[ [ 3)) [ [ 1))
[1) "own"

[ [3)) [ [2)]
[1) "rent"

[ [4)]
[1] 1 2 3 4 5

[ [5)]
 3.1 Introduction toR

[I 1] [ 1 2] [ 13 J
[11 J 1 5
[21 J 3 0
[3 1 J 3 4

In displaying the contents of assortment, the use of the double brackets, [ [] ] , is of particular
importance. As the following Rcode illustrates, the use of the single set of brackets only accesses an item
in the list, not its content.
# examine the fifth object, loll in the list
class(assortment[S]) .. returns "[Link]"
..
tt

length(assortment[S]) tt returns 1

class(assortment[[S]]) # returns "matrix"

length(assortment[[S]]) # returns 9 {for the 3x3 matrix)

As presented earlier in the data frame discussion, the s tr ( ) function offers details about the structure
of a list.
str(assortment)
List of 5
$ : chr "football"
$ : num 705
$ :List of 2

$
0 $ : chr "own "
0

00$ : chr "rent"

TRACE KTU
int [1: 5] 1 2 3 4 5
$ : num [1: 3 1 : 3] 1 3 3 5 0 4 3 3 3
1

Factors
Factors were briefly introduced during the discussion of the gender variable in the data frame sales.
In this case, gender could assume one of two levels: ForM. Factors can be ordered or not ordered. In the
case of gender, the levels are not ordered.
class(sales$gender) # returns "factor"
[Link](sales$gender) # returns FALSE

Included with the ggplot2 package, the diamonds data frame contains three ordered factors.
Examining the cut factor, there are five levels in order of improving cut: Fair, Good, Very Good, Premium,
and Ideal. Thus, sales$gender contains nominal data, and diamonds$cut contains ordinal data.
head(sales$gender) # display first six values and the levels

F F l-1 1'-1 F F
Levels: F l\1

library(ggplot2)
data(diamonds) # load the data frame into the R workspace
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

str(diamonds)
'[Link]': 53940 obs. of 10 variables:
$ carat num 0.23 0.21 0.23 0.29 0.31 0.24 0.24 0.26 0.22 ...
$ cut [Link] w/ 5 levels "Fair"c"Good"c .. : 5 4 2 4 2 3 ...
$ color [Link] w/ 7 levels "D"c"E"c"F"c"G"c .. : 2 2 2 6 7 7
$ clarity: [Link] w/ 8 levels "I1"c"SI2"c"SI1"< .. : 2 3 5 4 2
$ depth num 61.5 59.8 56.9 62.4 63.3 62.8 62.3 61.9 65.1 59.4
$ table num 55 61 65 58 58 57 57 55 61 61 ...
$ price int 326 326 327 334 335 336 336 337 337 338
$ X num 3.95 3.89 4.05 4.2 4.34 3.94 3.95 4.07 3.87 4 ...
$ y num 3.98 3.84 4.07 4.23 4.35 3.96 3.98 4.11 3.78 4.05
$ z num 2.43 2.31 2.31 2.63 2.75 2.48 2.47 2.53 2.49 2.39

head(diamonds$cut) # display first six values and the levels

Ideal Premium Good Premium Good Very Good
Levels: Fair c Good c Very Good < Premium < Ideal

Suppose it is decided to categorize sales$sales_ totals into three groups-small, medium,

and big-according to the amount of the sales with the following code. These groupings are the basis for
the new ordinal factor, spender, with levels {small, medium, big}.
# build an empty character vector of the same length as sales
sales_group <- vector (mode=''character",

TRACE KTU length=length(sales$sales_total))

# group the customers according to the sales amount

sales_group[sales$sales_total<100] <- "small"
sales_group[sales$sales_total>=100 & sales$sales_total<500] <- "medium"
sales_group[sales$sales_total>=500] <- "big"

# create and add the ordered factor to the sales data frame
spender<- factor(sales_group,levels=c("small", "medium", "big"),
ordered = TRUE)
sales <- cbind(sales,spender)

str(sales$spender)
[Link] w/ 3 levels "small"c"medium"c .. : 3 2 1 2 3 1 1 1 2 1 ...

head(sales$spender)
big medium small medium big small
Levels: small < medium c big

The cbind () function is used to combine variables column-wise. The rbind () function is used
to combine datasets row-wise. The use of factors is important in several Rstatistical modeling functions,
such as analysis of variance, aov ( ) , presented later in this chapter, and the use of contingency tables,
discussed next.
 3.11ntrodudion toR

Contingency Tables
In R, table refers to aclass of objects used to store the observed counts across the factors for agiven dataset.
Such a table is commonly referred to as a contingency table and is the basis for performing a statistical
test on the independence of the factors used to build the table. The following Rcode builds acontingency
table based on the sales$gender and sales$ spender factors.
# build a contingency table based on the gender and spender factors
sales_table <- table{sales$gender,sales$spender)
sales_table
small medium big
F 1726 2746 563
M 1656 2723 586

class(sales_table) returns "table"

typeof(sales_table) returns "integer"
dim{sales_table) # returns 2 3

# performs a chi-squared test

summary(sales_table)
Number of cases in table: 10000
Number of factors: 2
Test for independence of all factors:

TRACE KTU
Chisq = 1.516, df = 2, p-value = 0.4686

Based on the observed counts in the table, the summary {) function performs a chi-squared test
on the independence of the two factors. Because the reported p-value is greater than 0.05, the assumed
independence ofthe two factors is not rejected. Hypothesis testing and p-values are covered in more detail
later in this chapter. Next, applying descriptive statistics in Ris examined.

3.1.4 Descriptive Statistics

It has already been shown that the summary () function provides several descriptive statistics, such as
the mean and median, about avariable such as the sales data frame. The results now include the counts
for the three levels of the spender variable based on the earlier examples involving factors.
summary(sales)
cust ·- id sales - total nurn --- of orders gende1· spender
!,lin. :100001 r~lin. 30.02 !\lin. 1.000 F:5035 small :3382
1st Qu. :102501 1st Qu.: 80.29 1st Qu.: 2.000 1<1:4965 medium:5469
!\led ian :105001 r<ledian : 151.65 1\ledian : 2.000 big : 114 9
!\lean :105001 r-lean 249.46 r'lean 2.428
3rd Qu. : 107500 3rd Qu.: 295.50 3rd Qu.: 3.000
!\lax. :110000 r~Iax. : 76 06 . 0 9 1~1ax. :22.000

The following code provides some common Rfunctions that include descriptive statistics. In parenthe-
ses, the comments describe the functions.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

:. [Link], assig::
x <- sales$sales_total
y <- sales$num_of_orders

cor(x,yl :: returns 0.75080!5 correlatlor.)

cov(x ,y l II returns 345.~111 (covarianc<?)
IQR(x) ff :retu::n~ :ns.:n ([Link] range)
mean(x) ,. returns 249.4';'i7 mean)
~

median(x) returns 151.65 (me:lianl

range (x ) te:urns 30.0::: 7 06 . 09 min rna:-:)
"
sd(x) ., returns '-l9.0508 ::;:... :1. :ie·:.
var(x) retur::s ~C17r.l.~ ··ari:'l!!~"",...

The IQR () function provides the difference between the third and the first quarti [Link] other fu nc-
tions are fairly self-explanatory by their names. The reader is encouraged to review the available help files
for acceptable inputs and possible options.
The function apply () is useful when the same function is to be applied to several variables in a data
frame. For example, the following Rcode calculates the standard deviation for the first three variables in
sales. In the code, setting MARGIN=2 specifies that the sd () function is applied over the columns.
Other functions, such as lapply () and sapply (), apply a function to a list or vector. Readerscan refer
to the Rhelp files to learn how to use these functions.

TRACE KTU
apply (sales[,c (l : 3) ], MARGIN=2, FUN=Sd )

Additional descriptive statistics can be applied wi th user-defined funct ions. The following Rcode
defines a function, my_ range () , to compute thedifference between the maximum and minimum va lues
returned by the range () function. In general, user-defined functions are usefu l for any task or operation
that needsto be frequently [Link] information on user-defined functions is available by entering
help ( 11 function 11 ) in the console.
# build a functi~n tv plvviJ~ the difterence bet~een
~ -he maxrmum and thE .:m • •.1<
my_ range < - function (v) {range (v ) (2] - range (v) [1)}
my_range (x )

3.2 Exploratory Data Analysis

So far, this chapter has addressed importing and exporting data in R, basic data types and operations, and
generating descriptive statistics. Functions such as summary () can help analysts ea sily get an idea of
th e magnitude and range of the data, but other aspects such as linear relationships and distributions are
more difficult to see from descriptive statistics. For example, the following code shows a summary view of
a data frame data with two columns x and y. The output shows the range of x and y, but it's not clear
what the relationship may be between these two variables.
 3.2 Exploratory Data Analysis

summary (data )
·.. y
M1n. : 1.90481 ~·1n .

1st Qu . : -0.66321 !s• 0u

Nedian : 0 . 0"'167 N•d.111
Nean 0.0-52~

3rd Qu .: 0 . 65414 r t ,,u

A useful way to detect patterns and anomalies in the data is through the exploratory dataanalysis with
visualization. Visualization gives a succinct, holistic view of the data that may be difficult to grasp from the
numbers and summaries alone. Variables x and y of the data frame data can instead be visual ized in a
scatterplot (Figure 3-5). which easily depicts the relationship between two variab les. An important facet
of the initial data exploration, visualization assesses data cleanliness and suggests potentially important
relationships in the data prior to the model planning and building phases.

Scatterplot of X and Y

'·

o· TRACE KTU
-1·

2·

2 0 2
X

FIGURE 3-5 A scatterplot can easily show if x andy share a relation

The code to generate data as well as Figure 3-5 is shown next.

x <- rno rm (SO)

y <- x + rnorm(SO , mea n=O , sd=O . S)

data<- as . data . f rame(cbind(x , y ) )

REVIEW OF BASIC DATA ANALYTIC METHODS USING R

s ummary (data )

library (ggplo t 2)
ggpl ot (data, aes (x=x , y=y)) +
geom_point (size=2) +
ggtitle ("Scatterplo t o f X and Y" ) +
theme ([Link]=el emen t_t ex t(s i ze= l 2) ,
axis. title el emen t_text (si ze= l4 ) ,
[Link] = e l ement_ t ex t(si ze=20 , fa ce ="bold" ))

Explo ra tory data analysis [9] is adata ana lysis approach to reveal the important characteristics of a
dataset, mainly through visualization. This section discusses how to use some basic visualization techniques
and the plotting feature in Rto perform exploratory data analysis.

3.2.1 Visualization Before Analysis

To illustrate the importance of visualizing data, consider Anscombe'squartet. Anscombe'squartet consists
of four datasets, as shown in Figure 3-6. It was constructed by statistician Francis Anscom be [10] in 1973
to demonstrate the importance of graphs in statistical analyses.

#1 #2 # 3 #4
X y X y X y X y

TRACE KTU
4 4.26 4 3 10 4 5.39 8 5 25
5 5.68 5 4 74 5 5.73 8 5.56
6 7.24 6 6 13 6 6.08 8 5.76
7 4.82 7 7.26 7 6.42 8 6.58
8 6.95 8 8. 14 6.77 8 6.89
9 8.81 9 8.77 9 7. 11 8 7.04
10 8.04 10 9. 14 10 7.46 8 7.7 1
11 8.33 11 9.26 11 7.81 8 7.91
12 10.84 12 9. 13 12 8. 15 8 8.47
13 7.58 13 8.74 13 12.74 8 8.84
14 9.96 14 8. 10 14 8.84 19 12.50

fiGURE 3-6 Anscombe's quartet

The four datasets in Anscombe'squartet have nearly identical statistical properties, as shown in Table 3-3.

TABLE 3-3 Statistical Properties ofAnscombe's Quartet

Statistical Property Value

Meanof x 9

Variance of y 11

M ean ofY 7.50 (to 2decimal points)

 3.2 Exploratory Data Analysis

Variance of Y 4.12 or4.13 (to 2 decimal points)

Correla tions between x andy 0.816

Linear regression line y =3.00 + [Link] (to 2 decimal points)

Based on the nearly identical statistical properties across each dataset, one might conclude that these
four datasets are quite similar. However, the scatterplotsin Figure 3-7 tell a different story. Each dataset is
plotted asascatterplot, and the fitted lines are theresult of applying linear regression models. The estimated
regression line fits Dataset 1 reasonably well. Dataset 2 is definitely nonlinear. Dataset 3 exhibits a linear
trend, with one apparent outlier at x = 13. For Dataset 4, the regression line fits the dataset quite well.
However, with only points at two x values, it is not possible to determine that the linearity assumption is
proper.

~ I KTU~
12

TRACE
• • • • ••• ••

•
•
3 4

12
• •
• ••
:t 5
••• •

10 15
X
~:
5 10 15

FIGURE 3-7 Anscombe's quartet visualized as scatterplots

The Rcode for generating Figure 3-7 is shown next. It requires the Rpackage ggplot2 [11]. which can
be installed simply by running thecommand install . p ackages ( "ggp lot2" ) . The anscombe
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

dataset for the plot is included in the standard Rdistribution. Enter data ( ) for a list of datasets included
in the R base distribution. Enter data ( Da tase tName) to make a dataset available in the current
workspace.
In the code that follows, variable levels is created using the gl (} function, which generates
factors offour levels (1, 2, 3, and 4), each repeating 11 times. Variable myda ta is created using the
with (data, expression) function, which evaluates an expression in an environment con-
structed from da [Link] this example, the data is the anscombe dataset, which includes eight attributes:
xl, x2, x3, x4, yl, y2, y3, and y4. The expression part in the code creates a data frame from the
anscombe dataset, and it only includes three attributes: x, y, and the group each data point belongs
to (mygroup).
[Link](''ggplot2") # not required i f package has been installed

data (anscombe) It load the anscombe dataset into the current \'iOrkspace
anscombe
x1 x2 x3 x4 y1 y2 y3 y4
1 10 10 10 8 8. O·l 9.14 7.-16 6.58
2 8 8 8 8 6.95 8.14 6.77 5.76
13 13 13 7.58 8.74 12.74 7.71
4 9 9 8.81 8.77 7.11 8.84
5 11 11 11 8.33 9.26 7.81 8.·±7
6 14 14 14 8 9. 9G 8.10 8.34 7.04
7
8
9
10
6
4
12 12 12
7
6
4

7
TRACE KTU
6
4 19

7
8 7.24 6.13
·l. 26 3.10
8 10. 8•1 9.13
8 4.82 7.26
6. •J8

8.15
6.-12
5.25
5. 3 9 12.50
5.56
7.91
11 5 5 5 8 5.68 4.74 5.73 6.89

nrow(anscombe) It number of rows

[1] 11

# generates levels to indicate which group each data point belongs to

levels<- gl(4, nrow(anscombe))
levels
[1] 1 1 1 1 1 1 1 1 l l l 2 .; 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3
[ 34] 4 4 4 4 4 •l ·l 4 4 4 4
Levels: 1 2 3 4

# Group anscombe into a data frame

mydata <- with(anscombe, [Link](x=c(xl,x2,x3,x4), y=c(yl,y2,y3,y4),
mygroup=levels))

mydata
X y mygroup
10 8.04
2 8 6.95
13 7.58
4 9 8.81
 3.2 Exploratory Data Analysis

...
4,
1
B
1 ...
'i.S6
4

-l3 8 7 l 0 4
44 B 6 . 89 4

A Ma~f -~atterp "tF ~siny th ruplot~ package

library (ggplot2 )
therne_set (therne_bw ()) - s L rlot color :~erne

j\11 ' 1 7
ggplot (rnydata, aes (x, y )) +
geom_point (size=4 ) +
geom_srnooth (rnethod="lrn ", fill=NA, f ullrange=TRUE ) +
facet_wrap (-rnygroup )

3.2.2 Dirty Data

Thissection addresseshow dirty data ca n be detected in th edata expl oration phase with [Link]
general, analysts should look for anomalies, verify the data with domain knowledge, and decide the most
appropriate approach to clean the data.
Consider ascenario in which a bank isconducting data analyses of itsaccount holders to gauge customer
retention. Figure 3-8 shows the age distribution of the account holders.

TRACE KTU
0
...
0 -

>.
u
~~
~~
cQJ
:J
0"
QJ
u:
8 -J
~

0 -'

Age
FIGURE 3-8 Age distribution of bank account holders

If the age data isin a vector called age, the graph can be created with the following Rscript:

hist(age, br eaks=l OO , main= "Age Distributi on of Account Holders ",

xlab="Age", ylab="Frequency", col ="gray" )

The figure shows that the median age of the account holdersis around 40. A few accountswith account
holder age less than 10 are unusual but plausible. These could be custodial accounts or college savings
accounts set up by the parents of young children. These accountsshould be retained for future analyses.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

However, the left side of the graph shows a huge spike of customers who are zero years old or have
negative ages. This is likely to be evidence of missing data. One possible explanation is that the null age
values could have been replaced by 0 or negative values during the data input. Such an occurrence may
be caused by entering age in atext box that only allows numbers and does not accept empty values. Or it
might be caused by transferring data among several systems that have different definitions for null values
(such as NULL, NA, 0, -1, or-2). Therefore, data cleansing needs to be performed over the accounts with
abnormal age values. Analysts should take acloser look at the records to decide if the missing data should
be eliminated or if an appropriate age value can be determined using other available information for each
of the accounts.
In R, the is . na (} function provides tests for missing values. The following example creates avector
x where the fourth value is not available (NA). The is . na ( } function returns TRUE at each NA value
and FALSE otherwise.
X<- c(l, 2, 3, NA, 4)
[Link](x)
[1) FALSE FALSE FALSE TRUE FALSE

Some arithmetic functions, such as mean ( }, applied to data containing missing values can yield an
na. rm parameter to TRUE to remove the missing value during the
NA result. To prevent this, set the
function's execution.
mean(x)
[1) NA
mean(x, [Link]=TRUE)
[1) 2. 5 TRACE KTU
The na. exclude (} function returns the object with incomplete cases removed.
DF <- [Link](x = c(l, 2, 3), y = c(lO, 20, NA))
DF
X y
1 1 10
2 2 20
3 3 NA

DFl <- [Link](DF)

DFl
X y
1 1 10
2 2 20

Account holders older than 100 may be due to bad data caused by typos. Another possibility is that these
accounts may have been passed down to the heirs of the original account holders without being updated.
In this case, one needs to further examine the data and conduct data cleansing if necessary. The dirty data
could be simply removed or filtered out with an age threshold for future analyses. If removing records is
not an option, the analysts can look for patterns within the data and develop a set of heuristics to attack
the problem of dirty data. For example, wrong age values could be replaced with approximation based
on the nearest neighbor-the record that is the most similar to the record in question based on analyzing
the differences in all the other variables besides age.
 3.2 Exploratory Data Analysis

Figure 3-9 presents another example of dirty data. The distribution shown here corresponds to the age
of mortgages in a bank's home loan portfolio. The mortgage age is calculated by subtracting the origina-
tion date of the loan from the current date. The vertical axis corresponds to the number of mortgages at
each mortgage age.

Portfolio Distribution, Years Since Origination

0
0

"'
~

0
0
~
0
0
u>- (X)
cQj
0
::J 0
cY <D
~
u. 0
..,.
0

0
0
"'
0
I
0 2 6 8 10
Mortgage Age

FIGURE 3-9

TRACE KTU
Distribution ofmortgage in years since origination from a bank's home loan portfolio

If the data is in a vector called mortgage, Figure 3-9 can be produced by the following Rscript.

hist (mortgage, breaks=lO, xlab='Mortgage Age ", col= "gray•,

main="Portfolio Distribution, Years Since Origination" )

Figure 3-9 shows that the loans are no more than 10 years old, and these 10-year-old loans have a
disproportionate frequency compared to the rest of the population. One possible explanation is that the
10-year-old loans do not only include loans originated 10 years ago, but also those originated earlier than
that. In other words, the 10 in the x-axis actually means"<! 10. This sometimes happens when data is ported
from one system to another or because the data provider decided, for some reason, not to distinguish loans
that are more than 10 years old. Analysts need to study the data further and decide the most appropriate
way to perform data cleansing.
Data analysts shou ld perform sanity checks against domain knowledge and decide if the dirty data
needs to be eliminated. Consider the task to find out the probability of mortga ge loan default. If the
past observations suggest that most defaults occur before about the 4th year and 10-year-old mortgages
rarely default, it may be safe to eliminate the dirty data and assume that the defaulted loans are less than
10 years old. For other ana lyses, it may become necessary to track down the source and find out the true
origination dates.
Dirty data can occur due to acts of [Link] the sales data used at the beginning of this chapter,
it was seen that the minimum number of orders was 1 and the minimum annual sales amount was $30.02.
Thus, there isa strong possibility that the provided dataset did not include the sales data on all customers,
just the customers who purchased something during the past year.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

3.2.3 Visualizing a Single Variable

Using visual representations of data is a hallmark of exploratory data analyses: letting the data speak to
its audience rather than imposing an interpretation on the data a priori. Sections 3.2.3 and 3.2.4 examine
ways of displaying data to help explain the underlying distributions of a single variable or the relationships
of two or more variables.
R has many fu nctions avai lable to examine a single variable. Some of these func tions are listed in
Table 3-4.

TABLE 3-4 Example Functions for Visualizing a Single Variable

Function Purpose
p l o t (data ) Scatterplot where x is the index andy is the value;
suitable for low-volume data

barp lot (data ) Barplot with vertical or horizontal bars

dotchart (data ) Cleveland dot plot [12)

hist (data ) Histogram

plot(density (data )) Density p lot (a continuous histogram)

s tem (data)

rug (data ) TRACE KTU Stem-and -leaf plot

Add a rug representation (1-d plot) of the data to an

existing plot

Dotchart and Barplot

Dotcha rt and barplot portray continuous values with labels from a discrete variable. A dotchart can be
created in Rwith the function dot cha rt ( x , lab e l= ... ) , where x is a numeric vector and l a bel
is a vector of categorical labels for x. A barplot can be created with the barplot (h e igh t ) function,
w here h eigh t represents a vector or matrix. Figure 3-10 shows (a) a dotchart and (b) a barplot based
on the mtcars dataset, which includes the fuel consumption and 10 aspects of automobile design and
performance of 32 automobiles. This dataset comes with the standard Rdistribution.
The plots in Figure 3-10 can be produced with the following Rcode.

data (mtcars )
dotchart (mtcars$mpg,labels=row . names (mtcars ) ,cex=.7,
main= "Mi les Per Gallon (MPG ) of Car Models",
xlab ="MPG" )
barplot (tabl e (mtcars$cyl ) , main="Distribu:ion of Car Cyl inder Counts",
x lab= "Number of Cylinders" )

Histogram and Density Plot

Figure 3-ll(a) includ es a histogram of household income. The histogram shows a clear concentration of
low household incom es on the left and the long tail of the higher incomes on the right.
 3.2 Explorat ory Dat a Analysis

[Link] Per Gallon ([Link]) of Cor Models

Volvo U2f 0

Uastreb Bota 0

Ferran [)n)
Ford Panttra L
Lotus Europa
Pot3che 91 • -2 0

F'"1X1·9 0

Ponbac Frebrd Distribution of Car Cylinder counts

ComaroZ28
AJ.ICJavtlw1 ~
Dodge Chalenger
Toyota Corona
Toyota Carob 0 ~
Hondt CNC 0
0
F,.1 128
~
Chrysltr ~nal 0
L11cok1 Cont11tnt.l
C.d.. c Fleotwoo4 o co
Utrc •54Slt
Utrc 4SOSL
"'

D
[Link]. 0

Utrc2!0(
Were 280 ....
Wtre230
llerc 2•00
Ousttr 360
Vttant 0

Homtt SportabOU1 0
Homtl' Drrve
Datsun 7 10 6 8
Uazdo R.X' Wag
Uazda RXI 0 Ntmler ol Cylinders

TRACE KTU
10 15 20 2S 30

UPG

(a) (b)
FIGURE 3-10 (a) Dotchart on the miles per gallon of cars and (b) Barplot on the distribution ofcar cylind er
counts

Histogram of Income Distribution of Income (log10 scale)

....
0
"
N
0

"' 0
0

"'
..
i';
c:
:>
.,
0

"'
?;-
v;
.,
c:
00
0

CT

u: "'
0
0 "'
0

0
N "'
0

N
~ 0

0
0 0

Oe•OO 1e+05 2e+05 3e+05 4e•05 5e• 05 40 45 50 55

Income N = 4000 Band\Yidlh = 0 02069

FIGURE 3·11 (a) Histogram and (b) Density plot of household income
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

Figure 3-11 (b) shows a density plot of the logarithm of household income values, which emphasizes
the distribution. The income distribution is concentrated in the center portion of the graph. The code to
generate the two plots in Figure 3-11 is provided next. The rug ( } function creates a one-dimensional
density plot on the bottom of the graph to emphasize the distribution of the observation.
# randomly generate 4000 observations from the log normal distribution
income<- rlnorm(4000, meanlog = 4, sdlog = 0.7)
summary (income)
Min. 1st Qu. [Link] t>!ean 3rd Qu. f.!ax.
4.301 33.720 54.970 70.320 88.800 659.800
income <- lOOO*income
summary (income)
Min. 1st Qu. f.!edian f.!ean 3rd Qu. 1\!ax.
4301 33720 54970 70320 88800 659800
# plot the histogram
hist(income, breaks=SOO, xlab="Income", main="Histogram of Income")
# density plot
plot(density(loglO(income), adjust=O.S),
main="Distribution of Income (loglO scale)")
# add rug to the density plot
rug(loglO(income))

In the data preparation phase of the Data Analytics Lifecycle, the data range and distribution can be

TRACE KTU
obtained. If the data is skewed, viewing the logarithm of the data (if it's all positive) can help detect struc-
tures that might otherwise be overlooked in a graph with a regular, nonlogarithmic scale.
When preparing the data, one should look for signs of dirty data, as explained in the previous section.
Examining if the data is unimodal or multimodal will give an idea of how many distinct populations with
different behavior patterns might be mixed into the overall population. Many modeling techniques assume
that the data follows a normal distribution. Therefore, it is important to know if the available dataset can
match that assumption before applying any of those modeling techniques.
Consider a density plot of diamond prices (in USD). Figure 3-12(a) contains two density plots for pre-
mium and ideal cuts of diamonds. The group of premium cuts is shown in red, and the group of ideal cuts
is shown in blue. The range of diamond prices is wide-in this case ranging from around $300 to almost
$20,000. Extreme values are typical of monetary data such as income, customer value, tax liabilities, and
bank account sizes.
Figure 3-12(b) shows more detail of the diamond prices than Figure 3-12(a) by taking the logarithm. The
two humps in the premium cut represent two distinct groups of diamond prices: One group centers around
log10 price= 2.9 (where the price is about $794), and the other centers around log 10 price= 3.7 (where the
price is about $5,012). The ideal cut contains three humps, centering around 2.9, 3.3, and 3.7 respectively.
The Rscript to generate the plots in Figure 3-12 is shown next. The diamonds dataset comes with
the ggplot2 package.
library("ggplot2")
data(diamonds) # load the diamonds dataset from ggplot2

# Only keep the premium and ideal cuts of diamonds

 3.2 Exploratory Data Analysis

niceDiamonds <- diamonds [diamonds$cut=="Pr emi um" I

diamonds$cut== " Ide a l •, I

summary(niceDiamonds$cut )
Pr m1u !:! a ..
0 0 137<ll . lSSl

# plot density plot of diamond prices

ggplo t( niceDiamonds, ae s(x=price , fill =cut) ) +
geom_density(alpha = .3, col or=NA)

# plot density plot of the loglO of diamond prices

ggpl ot (niceDiamonds , aes (x=logl O(price) , f il l =cut ) ) +
geom_density (alpha = . 3, color=NA)

As an alternative to ggplot2, the lattice package provides a function ca lled densityplot ()

for making simple density plot s.

.. '

~
o;
;
3t
'

'.
TRACE KTU ~

..
;;
c
cut
Premium
Jcsu•
" "

It
'

0 0

1~300
pri ce togtO(prtce)

(a) (b)

FIGURE 3-12 Density plot s of (a) d iamond prices and (b) t he logarit hm ofdiamond p rices

3.2.4 Examining Multiple Variables

A scatterp lot (shown previously in Figure 3-1 and Figure 3-5) is a simple and w idely used visualization
for findin g the relationship among multiple va riables. A sca tterplo t ca n represent data with up to fi ve
variables using x-axis, y-axis, size, color, and shape. But usually only t wo to four variables are portrayed
in a scatterplot to minimize confusion. When examining a scatterplot, one needs to pay close attention
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

to the possible relationship between the variables. If the functiona l relationship between the variables is
somewhat pronounced, the data may roughly lie along a straight line,a parabola, or an exponential curve.
If variable y is related exponentially to x, then the plot of x versus log (y) is approximately linea r. If the
plot looks more like acluster without apattern, the corresponding variables may have aweak relationship.
The scatterplot in Figure 3-13 portrays the relationship of two variables: x and y . The red line shown
on the graph is the fitted line from the linear regression. Linear regression wi ll be revisited in Chapter 6,
"Advanced Analytical Theory and Methods: Reg ression." Figure 3-13 shows that the regression line does
not fit the data well. This is a case in which linear regression cannot model the relationship between the
variables. Alternative methods such as the l oess () functio n ca n be used to fit a nonlinear line to the
data. The blue curve shown on the graph represents the LOESS curve, which fits the data better than linear
regression.

0
0
0
0
N

.,.,
0

0
0

TRACE KTU
.,.,
0

0
0
0 o o

0
0

0
0
0

0 2 4 6 8 10

FIGURE 3-13 Examining two variables with regression

The Rcode to produce Figure 3-13 is as follows. The runi f ( 7 5, 0 , 1 0) generates 75 numbers
between 0 to 10 with random deviates, and the numbers conform to the uniform distribution. The
r norm( 7 5 , o , 2o) generates 75 numbers that conform to the normal distribu tion, with the mean eq ual
to 0 and the standard deviation equal to 20. The poi n ts () function is a generic function that draws a
sequence of points at the specified coordinates. Parameter type=" 1" tells the function to draw a solid
line. The col parameter sets the color of the line, where 2 represents the red color and 4 represents the
blue co lor.

7 numbers ben:een ~nd 10 ~f unifor~ distribution

x c- runif(75 , 0 , 1 0)

x c- sort(x)
y c - 200 + xA 3 - 10 * x A2 + x + rnorm(75, 0 , 20)

lr c- lm(y - x) ::a_l_ .eqL .;~l ..

poly c- loess( y - x) L E5~
 3.2 Exploratory Data Analysis

fit <- predict (pol y) fit a nonlinear lice

plot (x, y )

# draw the fitted li~e for the l i near regression

points (x, lr$coeffic i e nts [l ] + lr$coeffic i ents [2 ] • x ,
type= " 1 ", col = 2 )

po ints (x, fit, type = "1" , col = 4 )

Dotchart and Barplot

Dotchart and barplot from the previous section can visualize multiple variables. Both of them use color as
an additional dimension for visualizing the data.
For the same mtcars dataset, Figure 3-14 shows a dotchart that groups vehicle cyl inders at they-axis
and uses colors to distinguish different cylinders. The vehicles are sorted according to their MPG values.
The code to generate Figure 3-14 isshown next.

Miles Per Gallon (MPG) of Car Models

Grouped by Cylinder

4
Toyota Corolla
Fl8t 128
Lotus Eu ropa
Honda Civic
F1at X1-9
Porscl1e 914- 2
TRACE KTU 0
0
0
0
0
0

t.terc2400 0
Mere 230 0
Datsun 710 0
Toyota Corona 0
Volvo 142E 0

6
Hornet 4 Drive 0
Mazda RX4 Wag 0
Mazda RX4 0
Ferrari Dine 0
I.! ere 280 0
Valiant 0
I.! ere 280C 0

8
Pontiac Firebird 0
Hornet Sporta bout 0
Mer e 450SL 0
Mere 450SE 0
Ford Pantera L 0
Dodge Challenger 0
AMC Javeun 0
Mere 450SLC 0
1.1aserati Bora 0
Chrysler Imperial 0
Duster 360 0
Camaro Z28 0
Lincoln Continental 0
Cadillac Fleet wood 0

I I I I

10 15 20 25 30

Miles Per Gallon

FIGURE 3-14 Dotplot to visualize multip le variables

REVI EW OF BASIC DATA ANALYTIC METHODS USING R

;: sor- bJ' mpg

cars <- mtcars[or der {mtcars$mpg ) , )

h grouping variab l e must be a factol

cars$cyl < - f actor {cars$cyl )

cars$col or[car s$cyl ==4) <- "red"

cars$color[ cars$cyl== 6) < - "blue •
ca r s$color[cars$cyl==8) < - "darkgreen •

dotchart {cars$mp g, labels=[Link]{cars ) , cex - . 7, groups= cars$cyl,

main=" Mi les Per Gallon {MPG ) of Car Mode l s \ nGr ouped by Cylinder• ,
xl ab="Mil es Per Gal l on•, co l or=cars$color, gcolor="bl ac k")

The barplot in Figure 3-15 visualizes the distri bution of car cyli nder counts and number of gears. The
x-axis represents the number of cylinders, and the color represents the number of gears. The code to
generate Figure 3-15 is shown next.

Distribution of Car Cylinder Counts and Gears

~
Number of Gears
• 3
~

TRACE KTU
4
!;!
D 5
CD

~
:J <D
0
u

""
N

4 6 8

Number of Cylinders

FIGURE 3-15 Barplot to visualize multiple variables

count s <- t able {mtcars$gear , mtcars$cyl )

barpl ot (counts, main= "Di s tributi on o f Car Cylinder Coun ts and Gears • ,
x l ab="Number of Cylinders • , ylab="Counts •,
col=c (" #OOO OFFFF" , "# 0080FFFF", "#OOFFFFFF") ,
legend = rownames (counts ) , beside- TRUE,
args. l egend = list (x= "top", title= "Number of Gears" ))
 3 .2 Exploratory Data Analysis

Box-and-Whisker Plot
Box-and-whisker plots show the distribution of a continuous variable for each value o f a discrete variable.
The box-and-whisker plot in Figure 3-16 visualizes mean household incomes as a function of region in
th e United States. The first digit of the U.S. postal ("ZIP") code corresponds to a geographical region
in the United States. In Figure 3-16, each data point corresponds to the mean household income from a
particular zip code. The horizontal axis represents the first digit of a zip code, ranging from 0 to 9, where
0 corresponds to the northeast reg ion ofthe United States (such as Maine, Vermont, and Massachusetts),
and 9 corresponds to the southwest region (such as Ca lifornia and Hawaii). The vertical axis rep resents
the logarithm of mean household incomes. Th e loga rithm is take n to bet ter visualize the distr ibution
of th e mean household incomes.

Mean Household Income by Zip Code

so-
..,
E
0

TRACE KTU
u
c
;:;
0
.c
'""'::J
0
J:
iii ~ 5- •
'"
:::!!
0
Cl
2

' ' ' '

2 5 6 8 9
Zlp1

FIGURE 3-16 A box-and-whisker plot of mean household income and geographical region

In this figure, the scatterplot is displayed beneath the box-and-whisker plot, with some jittering for the
overlap points so that each line of points widens into a strip. The "box" of the box-and-whisker shows t he
range that contains the central 50% of the data, and the line inside the box is the location of the median
value. The upper and lower hinges of the boxes correspond to the first and third quartiles of the data. The
upper whisker extends from the hinge to the highest value that is within 1.5 * IQR of the hinge. The lower
whisker extends from the hinge to the lowest value w ithin 1.5 * IQR of the hinge. IQR is the inter-qua rtile
range, as discussed in Section 3.1.4. The points outside the wh iskers can be considered possible outliers.
REVIEW OF BAS IC DATA ANALYTIC M ETHODS USING R

The graph shows how household income varies by reg ion. The highest median incomes are in region
0 and region 9. Region 0 is slightly higher, but the boxes for the two regions overlap enough that the dif-
ference between the two regions probably is not significant. The lowest household incomes tend to be in
region 7, which includes states such as Louisiana, Arka nsas, and Oklahoma.
Assuming adata frame called DF contains two columns (MeanHousehol dincome and Zipl), the
following Rscript uses the ggplot2 1ibrary [11 ] to plot a graph that is similar to Figure 3-16.

library (ggplot2 )
plot the jittered scat-erplot w/ boxplot
H color -code points with z1p codes
h th~ outlier . [Link] pr~vents the boxplot from p:c-•inq •h~ uutlier

ggplot (data=DF, aes (x=as . factor (Zipl ) , y=loglO(MeanHouseholdincome) )) +

geom_point(aes(color=factor (Zipl )) , alpha= 0 .2, pos it i on="j itter") +
geom_boxpl ot(outlier .size=O, alpha=O . l ) +
guides(colour=FALSE) +
ggtitle ("Mean Hous ehold Income by Zip Code")

Alternatively, one can create a simple box-and-whisker plot with the boxplot () function provided
by the Rbase package.

Hexbinplot for Large Datasets

TRACE KTU
This chapter ha s shown that scat terplot as a popular visualization can visualize data containing one or
more variables. But one should beca reful about using it on high-volumedata. lf there is too much data, the
structure of thedata may become difficult to see in [Link] acase to compare the logarithm
of household income against the yearsof education, as shown in Figure 3-17. The cluster in the scatterplot
on the left (a) suggestsa somewhat linear relationship of the two variables. However, onecannot rea lly see
the structure of how the data is distributed inside the cluster. This is a Big Data type of problem. Millions
or billions of data points would require different approaches for exploration, visualization, and analysis.

j
g] Counts
71&8
6328
SS22
8
c
'0
0
C!
</) 0
0
0
0 I" ··- 4 77 1
407S
34
! 8
0
~ 1&15
316
:;)
0 </) 8 0 fa u .J 1640
I

I
c <i 141 8
1051
IB ~ 739
::!i
0
0. 0
0
r .... 432
279
C! 132
.2
" 39
0 0
10

~'•W~.Eduauon
.. 1

0 5 10 15

MeanEduca1ion

(a) (b)

FIGURE 3-17 (a) Scatterplot and (b) Hexbinplot of household income against years ofeducation
 3.2 Exploratory Data Analysis

Although color and transparency can be used in a scatterplot to address this issue, a hexbinplot is
sometimes a better alternative. A hexbinplot combines the ideas of scatterplot and histogram. Similar to
a scatterplot, ahexbinplot visualizes data in the x-axis andy-axis. Data is placed into hexbins, and the third
dimension uses shading to represent the concentration of data in each hexbin.
In Figure 3-17(b), the same data is plotted using a hexbinplot. The hexbinplot shows that the data is
more densely clustered in astreak that runs through the center ofthe cluster, roughly along the regression
line. The biggest concentration is around 12 years of education, extending to about 15 years.
In Figure 3-17, note the outlier data at MeanEducation=O. These data points may correspond to
some missing data that needs further cleansing.
Assuming the two variables MeanHouseholdincome and MeanEduca tion are from a data
frame named zeta, the scatterplot of Figure 3-17(a) is plotted by the following Rcode.
# plot the data points
plot(loglO(MeanHouseholdincome) - MeanEducation, data=zcta)
# add a straight fitted line of the linear regression
abline(lm(loglO(MeanHouseholdincome) - MeanEducation, data=zcta), col='red')

Using the zeta data frame, the hexbinplot of Figure 3-17(b) is plotted by the following Rcode.
Running the code requires the use ofthe hexbin package, which can be installed by running ins tall
.packages ( "hexbin").
library(hexbin)

TRACE KTU
# "g" adds the grid, "r" adds the regression line
# sqrt transform on the count gives more dynamic range to the shading
# inv provides the inverse transformation function of trans
hexbinplot(loglO(MeanHouseholdincome) - MeanEducation,
data=zcta, trans= sqrt, inv = function(x) x... 2, type=c( 11 g 11 , 11
r 11 ) )

Scatterplot Matrix
A scatterplot matrix shows many scatterplots in a compact, side-by-side fashion. The scatterplot matrix,
therefore, can visually represent multiple attributes of a dataset to explore their relationships, magnify
differences, and disclose hidden patterns.
Fisher's iris dataset [13] includes the measurements in centimeters ofthe sepal length, sepal width,
petal length, and petal width for 50 flowers from three species of iris. The three species are setosa, versicolor,
and virginica. The iris dataset comes with the standard Rdistribution.
In Figure 3-18, all the variables of Fisher's iris dataset (sepal length, sepal width, petal length, and
petal width) are compared in ascatterplot matrix. The three different colors represent three species of iris
flowers. The scatterplot matrix in Figure 3-18 allows its viewers to compare the differences across the iris
species for any pairs of attributes.
REVIEW O F BA SIC DATA ANA LYTIC M ETHODS USIN G R

Fisher's Iris Dataset

..... w.4"t~
.....
20 25 30 35 •o 0510152025

••• I ..... "' ....

~- ..
~· f· . .."'
[Link] t' "' "' 0&<4

...... ..;.;.:.··
• ••• •
..... ,. .• "''"
14 11
10>1
'19
~

.. ...
11t
I ll

f.·.
)9

" . ..
;~·:
I

• •
•..,.··~til!~-= [Link] ...
~-· .
.,.. .
..,
Q

0 _ic
"'
~ .. - ~·
•• •
·.t.*
•. .
• =:t • • Petal. length

Petal. Width
"'
Q

FIGURE 3·18
H

TRACE KTU
55 65 75

• setosa
Scatterplot matrix ofFisher's {13] iris dataset
D verstcolor •
12 3<567

virgimca

Consider the scatterplot from the first row and third col umn of Figure 3-18, where sepal length iscom-
pared against petal [Link] horizontal axisis the petal length, and the vertical axis isthe sepal length.
The scatterplot shows that versicolor and virginica share similar sepal and petal lengths, although thelatter
has longer petals. The petal lengthsof all setosa are about the sa me, and the petal lengths are remarkably
shorter than the other two species. The scatterplot shows that for versicolor and virgin ica, sepal length
grows linearly with the petal length.
The Rcode for generating the scatterplot mat rix isprovided next.

I; define the colors

colors<- C( 11 red 11 11
green 11 ,
J
11
blue•')

~ draw the plot ma:rix

pairs(iris[l : 4], main= "Fisher ' s Iris Datase t•,

pch = 21, bg = colors[unclass ( iris$Species)]

= ~Qr qrdp~ica: pa~a~ :e~· - cl~!' p!ot - 1~9 :c :te ~1gure ~~a1o~

par (xpd = TRUE )

" ada l<"go::d

legend ( 0.2, 0 . 02, horiz = TRUE, [Link] (unique ( iris$Species )) ,
fil l = colors, bty = "n" )
 3.2 Explorat ory Data Analysis

The vector colors defines th e colo r sc heme for the plot. It could be changed to something like
colors<- c("gray50", "white" , " black " } to makethescatterplotsgrayscale.

Analyzing a Variable over Time

Visua lizing a variable over time is the same as visualizing any pair of variables, but in this case the goal is
to identify time-specific patterns.
Figure 3-19 plots the mon thly total numbers of international airline passengers (in thousands) from
January 1940 to December 1960. Enter plot (AirPassengers} in the Rconsole to obtain a similar
[Link] plot shows that, for each year, alarge peak occurs mid-year around July and August, and asma ll
peak happensaround the end of the year, possibly due to the holidays. Such a phenomenon isreferred to
as a seasonality effect.

0
0
CD

0
0
II'>

"'Q; 0
0>
.,c ....
0

"'"'
"'
Q, 0
0
< (")

TRACE KTU
0
0
N

0
~

1950 1952 1954 1956 1958 1960

Tune

FIGURE 3-19 Airline passenger counts from 1949 to 1960

Additionally, the overall trend is that the number of air passengers steadily increased from 1949 to
1960. Chapter 8, "Advanced Analytica l Theory and Methods: Time Series Analysis," discusses the analysis
of such datasets in greater detail.

3.2.5 Data Exploration Versus Presentation

Using visualization for data exploration is different from presenting results to stakeholders. Not every type
of plot issuitable for all audiences. Most ofthe plots presented earlier try to detail the data as clearly aspos-
sible for data scientists to identify structures and relationships. These graphs are more technical in nature
and are better suited to technical audiences such as data scientists. Nontechnical sta keholders, however,
generally prefer simple, clear graphics that focus on the message rather than the data.
Figure 3-20 shows the density plot on the distribution of account va lues from a bank. The data has been
converted to the log 10 [Link] plot includesa rug on the bottom to show the distribution of the variable.
This graph is more suitable for data scientists and business analysts because it provides information that
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

can be relevant to the downstream analysis. The graph shows that the transformed account values follow
an approximate normal distribution, in the range from $100 to $10,000,000. The median account value is
approximately $30,000 (1 o4s), with the majority of the accounts between $1,000 (1 03) and $1,000,000 (1 06).

Distribution of Account Values (log10 scale)

CD
ci
II)
ci
oq:
0
~
·a; ("')
c c::)
cu
0
N
c::)

.....
c::)

0
c::)

2 3 4 5 6 7

N = 5000 Bandwidth= 0.05759

FIGURE 3-20 Density plots are better to show to data scientists

TRACE KTU
Density plots are fairly technical, and they contain so much information that they would be difficult to
explain to less technical stakeholders. For example, it would be challenging to explain why the account
values are in the log 10 scale, and such information is not relevant to stakeholders. The same message can
be conveyed by partitioning the data into log-like bins and presenting it as a histogram. As can be seen in
Figure 3-21, the bulk of the accounts are in the S1,000-1,000,000 range, with the peak concentration in the
$10-SOK range, extending to $500K. This portrayal gives the stakeholders a better sense of the customer
base than the density plot shown in Figure 3-20.
Note that the bin sizes should be carefully chosen to avoid distortion of the [Link] this example, the bins
in Figure 3-21 are chosen based on observations from the density plot in Figure 3-20. Without the density
plot, the peak concentration might be just due to the somewhat arbitrary appearing choices for the bin sizes.
This simple example addresses the different needs of two groups of audience: analysts and stakehold-
ers. Chapter 12, "The Endgame, or Putting It All Together," further discusses the best practices of delivering
presentations to these two groups.
Following is the Rcode to generate the plots in Figure 3-20 and Figure 3-21.
# Generate random log normal income data
income= rlnorm(SOOO, meanlog=log(40000), sdlog=log(S))

# Part I: Create the density plot

plot(density(loglO(income), adjust=O.S),
main= 11 Distribution of Account Values (loglO scale)")
# Add rug to the density plot
 3.3 Statistical Methods for Evaluation

r ug (logl O(income))

l• :l... "1 ! .... < bl:-;.5''

breaks = c(O, 1000, 5000, 10000, 50000, 100000, SeS, le6, 2e7 )
"'! 1.:: . . ... ••' ,
bins = cut(income, breaks, include .lowest =T,
labels c ( "< lK", "1 - SK", "5- lOK" , "10 - SOK",
"50-lOOK" , "100 -S OOK" , "SOCK-1M", "> 1M") )
~ n r •L ri...
plot(bins, main "Dis tribut i on of Account Val ues ",
xl ab "Account value ($ USD) ",
ylab = "Number of Accounts", col= "blue ")

Distribution of Account Values

TRACE KTU
0

- <1K

FIGURE 3·21 Histograms are better to show to stakeholders

1-5K 5-IOK 10-50K 50- l OOK

AccOirl value (S USO)

100.500K --
500K-1M > 11.1

3.3 Statistical Methods for Evaluation

Visualization is useful for data exploration and presentation, but statistics is crucial because it may exist
throughout the entire Data Analytics Lifecycle. Statistical techniques are used during the initial data explo-
ration and data preparation, model building, evaluation of the final models, and assessment of how the
new models improve the situation when deployed in the field. In particular, statisticscan help answer the
following questions for data analytics:

• Model Building and Planning

• What are the best input variablesfor the model?

• Can the model predict the outcome given the input?

REVIEW OF BASIC DATA ANALYTIC METHODS USING R

• Model Evaluation

• Is the model accurate?

• Does the model perform better than an obvious guess?

• Does the model perform better than another cand idate model?

• Model Deployment

• Is the prediction sound?

• Does the model have the desired effect (such as reducing the cost}?

This sec tion discusses some useful statistical tools that may answer these questions.

3.3.1 Hypothesis Testing

When compari ng populations, such as testing or evaluating the difference of the means from two samples
of data (Figure 3-22}, a common technique to assess the difference or the significance of the difference is
hyp othesis testing.

TRACE KTU

FIGURE 3-22 Distributions of two samples ofdata

The basic concept of hypothesis testing is to form an assertion and test it with data. When perform-
in g hypothesis tests, the common assumption is that there is no difference between two samples. This
assumption is used as the default position for building the test or conducting a scientific experiment.
Statisticians refer to this as the null hyp o thesis (H0 ). The altern a tive hyp o thesis (H) is that there is a
 3.3 Statistical Methods for Evaluation

difference between two samples. For example, if the task is to identify the effect of drug A compared to
drug Bon patients, the null hypothesis and alternative hypothesis would be th is.

• fl 0: Drug A and drug Bhave the same effect on patients.

• fl A: Drug A has agreater effect than drug Bon patients.

If the task is to identify whether advertising Campaign Cis effective on reducing customer churn, the
null hypothesis and alternative hypothesiswou ld be as follows.

• fl0 : Campaign Cdoes not reduce customer churn better than the cu rrent campa ign method.
• flA: Campaign Cdoes reduce customer churn better than the current campa ign.
It is important to state the null hypothesis and alternative hypothesis, because misstating them is likely
to undermine the subsequent steps of the hypothesis testing process. A hypothesis test leads to either
rejecting the null hypothesis in favor of the alternative or not rejecting the null hypothesis.
Table 3·5 includes some examplesof null and alternative hypotheses that should be answered during
the analytic lifecycle.

TABLE 3-5 Example Null Hypotheses and Alternative Hypotheses

Application Null Hypothesis Alternative Hypothesis

Accuracy Forecast Model Xdoes notpredict better Model X predicts better than the existing

Engine
TRACE KTU
Recommendation
than the existing model.

Algorithm Y does notproduce

better recommendationsthan
model.

Algorithm Y produces better recommen·

dations than the current algorithm being
the current algorithm being used.
used.

Regression Thisvariable does not affect the This variable affects outcome because its
Modeling outcome because its coefficient coefficient isnot zero.
is zero.

Once a model isbuilt over the t raining data, it needs to be eva luated over the testing data to see if the
proposed model predicts better than the existing model curren tly being used. Th e null hypothesisis that
the proposed model does not predict better than the existing model. The alternative hypothesis is that
the proposed model indeed predicts better than the existing model. In accuracy forecast, the null model
could be that the sales of the next month are the same as the prior month. The hypothesis test needs to
evaluate if the proposed model provides a better prediction. Take a recommendation engine as an example.
The null hypothesis could be that the new algorithm does not produce better recommendations than the
current algorithm being deployed. The alternative hypothesis is that the new algorithm produces better
recommendations than the old algorithm.
When eva luating a model, sometimes it needs to be determined if agiven input variable improves the
model. In regression analysis (Chapter 6), for example, this is the same as asking if the regression coefficient
for a variable is zero. The null hypothesis is that the coefficient is zero, which means the variable does not
have an impact on the outcome. Thealternative hypothesis is that the coefficient is nonzero, which means
the variable does have an impact on the outcome.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

A common hypothesis test is to compare the means of two populations. Two such hypothesis test sare
discussed in Section 3.3.2.

3.3.2 Difference of Means

Hypothesis testing is a common approach to draw inferences on whether or not the two populations,
denoted pop1 and pop2, are different from each other. This section provides two hypothesistests to com-
pare the means of the respective populations based on sam ples randomly drawn from each population.
Specifically, the two hypothesis tests in this section consider the following null and alternative hypotheses.

• Ho: II , = ll2
• HA: II , ""' ll2

The 1', and 112 denote the population means of pop1 and pop2, respectively.
The basic testing approach is to compare the observed sample means, X,and X2, corresponding to each
population. If the values of X1 and X2 are approximately equal to each other, the distributions of X,and
X2 overlap substantially (Figure 3-23), and the null hypothesis is supported. A large observed difference
between the sample means indicates that the null hypothesis should be rejected. Formally, the difference
in means can be tested using Student's t-test or the Welch's t-test.

TRACE KTU Irx, '" :X2 .

this area is
large

FIGURE 3-23 Overlap ofthe two distributions is large if X1 ::::: X2

Student's t-test
Stud ent 's t- test ass umes that distributions of the t wo populations have equal but unknow n
variances. Suppose n1 and n2 samples are random ly and independently selected from two populations,
pop1 and pop2, respectively. If each population is normally distributed with the same mean (Jt 1 = Jt 2) and
wi th the sa me variance, then T (the t-statistic ), given in Equation 3-1, follows a t-distribution w ith
n, + n2 - 2 degrees of freedom (df).

where (3-1)
 3.3 Statistical Methods for Evaluation

The shape of the t-distribution is similar to the normal distribution. In fact, as the degrees of freedom
approaches 30 or more, the t-distribution is nearly identical to the normal distribution. Because the numera-
tor ofT is the difference of the sample means, if the observed value ofT is far enough from zero such that
the probability of observing such a value of Tis unlikely, one would reject the null hypothesis that the
population means are equal. Thus, for a small probability, say a= 0.05, T* is determined such that
P(ITI2: T*) = 0.05. After the samples are collected and the observed value ofT is calculated according to
Equation 3-1, the null hypothesis (p,1 = p 2) is rejected ifiTI2: r·.
In hypothesis testing, in general, the small probability, n, is known as the significance level of the test.
The significance level of the test is the probability of rejecting the null hypothesis, when the null hypothesis
is actually [Link] other words, for n = 0.05, if the means from the two populations are truly equal, then
in repeated random sampling, the observed magnitude ofT would only exceed r· 5% of the time.
In the following Rcode example, 10 observations are randomly selected from two normally distributed
populations and assigned to the variables x andy. The two populations have a mean of 100 and 105,
respectively, and a standard deviation equal to 5. Student's t-test is then conducted to determine if the
obtained random samples support the rejection of the null hypothesis.
# generate random observations from the two populations
x <- rnorm(lO, mean=lOO, sd=S) # normal distribution centered at 100
y <- rnorm(20, mean=lOS, sd=S) ll no~:mal distribution centered at 105

[Link](x, y, [Link]=TRUE) # run the Student's t-test

TRACE KTU
Two Sample t-test

data: x and y
t = -1.7828, df = 28, p-value = 0.08547
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-6.1611557 0.4271393
sample estimates:
mean of x mean of y
102.2136 105.0806

From the R output, the observed value of Tis t = -1.7828. The negative sign is due to the fact that the
sample mean of xis less than the sample mean of y. Using the qt () function in R, a Tvalue of 2.0484
corresponds to a 0.05 significance level.
# obtain t value for a two-sidec test at a 0.05 significance level
qt(p=[Link]/2, df=28, [Link]= FALSE)
2.048407

Because the magnitude of the observed T statistic is less than the T value corresponding to the 0.05
significance level Q-1.78281< 2.0484), the null hypothesis is not rejected. Because the alternative hypothesis
is that the means are not equal (p 1 :;z:: 11 2), the possibilities of both p, > 112 and p 1 < 11 2 need to be considered.
This form of Student's t-test is known as a two-sided hypothesis test, and it is necessary for the sum of the
probabilities under both tails of the t-distribution to equal the significance level. It is customary to evenly
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

divide the significance level between both tails. So, p = 0.05/2 = 0.025 was used in the qt () function to
obtain the appropriate t-value.
To simplify the comparison of the t-test results to the significance level, the Routput includes aquantity
known as the p -value. ln the preceding example, the p-value is 0.08547, which is the sum of P(T ~ - 1.7828)
and P(T ~ 1.7828). Figure 3-24 illustrates the t-statistic for the area under the tail of a t-distribution. The -t
and tare the observed values of the t-statistic. ln the Routput, t = 1.7828. The left shaded area corresponds
to the P(T ~ - 1.7828), and the right shaded area corresponds to the P(T ~ 1.7828).

-t 0
FIGURE 3-24 Area under the tails (shaded) ofa student's t-distribution

In the Routput, for a significance level of 0.05, the null hypothesiswould not be rejected because the
likelihood of a Tvalue of magnitude 1.7828 or greater would occur at higher probability than 0.05. However,
based on the p -value, if the significance level was chosen to be 0.10, instead of 0.05, the null hypothesis

TRACE KTU
would be rejected. In general, the p-value offers the probability of observing such a sample result given
the null hypothesis is TRUE.
A key assumption in using Student's t-test is that the population variances are equal. In the previous
example, the t . test ( ) function call includes var . equal=TRUE to specify that equality of the vari-
ances should be assumed. If that assumption is not appropriate, then Welch's t-test should be used.

Welch 's t-test

When the equal population variance assumption is not justified in performing Student's t-test for the dif-
ference of means, Welch's t-test [14] can be used based on T expressed in Equation 3-2.

(3-2)

where X,. 5,2, and n, correspond to the i-th sample mean, sample variance, and sample size. Notice that
Welch's t-test uses the sample va riance (5ll for each population instead of the pooled sample variance.
In Welch's test, under the remaining assumptions of random samples from two normal populations with
the same mean, thedistribution of Tisapproximated by the t-distribution. The following Rcode performs
the We lch's t-test on the same set of data analyzed in the earlier Student's t-test example.
 3.3 Statistical Methods for Evaluation

[Link](x, y, [Link]=FALSE) # run the Welch's t-test

l'lelch Two Sample t-test

data: x andy
t = -1.6596, df = 15.118, p-value = 0.1176
alternative hypothesis: true difference in neans is not equal to o
95 percent confidence interval:
-6.546629 0.812663
sample estimates:
mean of x mean of y
102.2136 105.0806

In this particular example of using Welch's t-test, the p-value is 0.1176, which is greater than the p-value
of 0.08547 observed in the Student's t-test example. In this case, the null hypothesis would not be rejected
at a 0.10 or 0.05 significance level.
It should be noted that the degrees of freedom calculation is not as straightforward as in the Student's
t-test. In fact, the degrees of freedom calculation often results in a non-integer value, as in this example.
The degrees of freedom for Welch's t-test is defined in Equation 3-3.

(3-3)

TRACE
df=l~r l~:r KTU --+--
n,-1 n2 -1
In both the Student's and Welch's t-test examples, the Routput provides 95% confidence intervals on
the difference of the means. In both examples, the confidence intervals straddle zero. Regardless of the
result of the hypothesis test, the confidence interval provides an interval estimate of the difference of the
population means, not just a point estimate.
A confidence interval is an interval estimate of a population parameter or characteristic based on
sample data. Aconfidence interval is used to indicate the uncertainty ofapoint [Link] is the estimate
of some unknown population mean f..L, the confidence interval provides an idea of how close xis to the
unknown p. For example, a 95% confidence interval for a population mean straddles the TRUE, but
unknown mean 95% of the time. Consider Figure 3-25 as an example. Assume the confidence level is 95%.
If the task is to estimate the mean of an unknown value Jt in a normal distribution with known standard
deviation u and the estimate based on n observations is x, then the interval ± ~ straddles the unknown
x
value of Jl with about a 95% chance. If one takes 100 different samples and computes the 95% confi-
dence interval for the mean, 95 ofthe 100 confidence intervals will be expected to straddle the population
mean Jt.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

FIGURE 3-25 A 95% confidence interval straddlin g the unknown population mean 1J

Confidence intervals appear again in Section 3.3.6 on ANOVA. Return ing to the discussion of hypoth-
es is testing, a key assumpti on in b oth t he Stud ent 's and Welch 's t-test is that the relevant population
attri bute is norma lly distributed. For non-normally dist ributed data, it is sometimes p ossible to transform
the co llected data to approx imate a normal distribution. For example, taki ng the logarithm of a d ataset

TRACE KTU
can often transform skewed d ata to a dataset that is at least symmetric arou nd its mean. Howeve r, if such
transformations are ineffective, there are tests like the Wi lcoxon ra nk-su m test that can be ap plied to see
if t wo population distributions are different.

3.3.3 Wilcoxon Rank-Sum Test

At-test represents a parametric test in that it makes assumptionsabout the population distributionsfrom
w hich th e sa mples are drawn. If the populations cann ot be assu med or transformed to follow a normal
distribution, a n onparametric test can be used . The Wilcoxo n rank-sum test [15] is a nonpa rametric
hypothesis test that checks w hether two populations are identically distributed. Assuming the two popula-
tions are identica lly distributed, one would expect that the ordering of any sampled observations would
be evenly intermixed among themselves. For example, in orderi ng the observations, one would not expect
to see a large number of observations from one population grouped together, especially at the beginning
or the end of ordering.
Let the t wo p opulations again be popl and pop2, w ith independently random samplesof size n1 and
n2 respectively. The total number of observations is then N = n1 + n2• The first step of the Wilcoxon test is
to rank the set of observations from the t wo groups as if they came from one la rge group. The smallest
observation receives a rank of 1, the second smallest observation receives a rank of 2, and so on with the
largest observation being assig ned the rank of N. Ties among the observations receive a ran k equal to
t he average of the ranks they span. The test uses ranks instead of numerical outcom es to avoid specific
assumptions about the shape of the distribution.
After ranking all the ob servations, the assig ned ranks are summed for at least one population's sample.
If the distribution of popl is shifted to the right of the other distribution, the rank-sum correspondi ng to
popl's sa mple shou ld be larger than the rank-sum of pop2. The Wilcoxon rank-sum test determines the
 3.3 Statistical Methods for Evaluation

significance of the observed rank-su ms. The following Rcode performs the test on the same dataset used
for the previous t-test.

[Link](x, y, [Link] TRUE)

:·:~.c :-c n r rJ.:

! 1 t

The wilcox. test ( l function ranks the observations, determines the respective rank-sums cor-
responding to each population's sample, and then determines the probability of such rank-sums of such
magnitude being observed assuming that the population distributions are identical. In this example, the
probability is given by the p-value of 0.04903. Thus, the null hypothesis would be rejected at a 0.05 sig-
nificance level. The reader is cautioned against interpreting that one hypothesis test is clearly better than
another test based solely on the examplesgiven in this section.

TRACE KTU
Because the Wilcoxon test does not assume anything about the population distribution, it isgenerally
considered more robust than the t-test. In other words, there are fewer assumptions to violate. However,
when it is reasonable to assume that the data is normally distributed, Student's or Welch's t-test is an
appropriate hypothesis test to consider.

3.3.4 Type I and Type II Errors

A hypothesis test may result in two types of errors, depending on whether the test accepts or rejects the
null hypothesis. These two errors are known as type I and type II errors.

• A type I error is the rejection of the null hypothesis when the null hypothesis is TRUE. The probabil-
ity of the type I error is denoted by the Greek letter n .

• A type II error is the acceptance of a null hypothesis when the null hypothesisis FALSE. The prob-
ability of the type II error is denoted by the Greek letter .1.

Table 3-61ists the four possible states of a hypothesis test, including the two types of errors.

TABLE 3-6 Type I and Type II Error

H0 is true H0 is false

H0 is accepted Correct outcome Type II Error

H0 is rejected Type/error Correct outcome

REVIEW OF BASIC DATA ANALYTIC METHODS USING R

The significance level, as mentioned in the Student's t-test discussion, is equivalent to the type I error.
For a significance level such as o = 0.05, if the null hypothesis (Jt 1= J1 1) is TRUE, there is a So/o chance that
the observed Tvalue based on the sample data will be large enough to reject the null hypothesis. By select-
ing an appropriate sig nificance level, the probability of commi tting a type I error can be defined before
any data is collected or analyzed.
The probability of committing aType II error is somewhat more difficult to determine. Iftwo population
means are truly not equal, the probability of committing a type II error will depend on how far apart the
means truly are. To reduce the probability of a type II error to a reasonable level, it is often necessary to
increase the sample size. This topic is addressed in the next section.

3.3.5 Power and Sample Size

The power of atest is the probability of correctly rejecting the null [Link] is denoted by 1- /.3, where
f] is the probability of a type II error. Because the power of a test improves as the sample size increases,
power is used to determine the necessary samplesize. In the difference of means, the power of ahypothesis
test depends on the truedifference of the population means. In other words, for a fixed significance level,
a larger sample size is required to detect a smaller difference in the means. In general, the magnitude of
the difference is known as the effect size. As the sample size becomes larger, it is easier to detect a given
effec t size, 6, as illustrated in Figure 3-26.

Moderate Sample Size Larger Sample Size

TRACE KTU
1------1 1------1
a' a'

F IGURE 3-26 A larger sample size better identifies a fixed effect size

With a large enough sample size, almost any effect size can appear statistically significant. However, a
very small effect size may be useless in a practical sense. It isimportan t to consider an appropriate effect
size for the problem at hand.

3.3.6ANOVA
The hypothesis tests presented in the previous sections are good for analyzing means between two popu-
lations. But what if there are more than two populations? Consider an example of testing the impact of
 3.3 Statistical Methods for Evaluation

nutrition and exercise on 60 candidates between age 18 and 50. The candidates are randomly split into six
groups, each assigned with a different weight loss strategy, and the goal is to determine which strategy
is the most effective.
o Group 1only eats junk food.
o Group 2only eats healthy food.
o Group 3eats junk food and does cardia exercise every other day.
o Group 4eats healthy food and does cardia exercise every other day.
o Group 5eats junk food and does both cardia and strength training every other day.

o Group 6eats healthy food and does both cardia and strength training every other day.

Multiple t-tests could be applied to each pair of weight loss strategies. In this example, the weight loss
of Group 1is compared with the weight loss of Group 2, 3, 4, 5, or 6. Similarly, the weight loss of Group 2is
compared with that of the next 4 groups. Therefore, a total of 15 t-tests would be performed.
However, multiplet-tests may not perform well on several populations for two reasons. First, because the
number oft-tests increases as the number of groups increases, analysis using the multiplet-tests becomes
cognitively more difficult. Second, by doing a greater number of analyses, the probability of committing
at least one type I error somewhere in the analysis greatly increases.
Analysis of Variance (ANOVA) is designed to address these issues. ANOVA is a generalization of the

TRACE KTU
hypothesis testing of the difference of two population means. ANOVA tests if any of the population means
differ from the other population means. The null hypothesis of ANOVA is that all the population means are
equal. The alternative hypothesis is that at least one pair of the population means is not equal. In other
words,
0 Ho:Jll = J12 = ··· = Jln
o HA: Jl; ::= J1i for at least one pair of i,j
As seen in Section 3.3.2, "Difference of Means," each population is assumed to be normally distributed
with the same variance.
The first thing to calculate for the ANOVA is the test statistic. Essentially, the goal is to test whether the
clusters formed by each population are more tightly grouped than the spread across all the populations.
Let the total number of populations be k. The total number of samples N is randomly split into the k
groups. The number of samples in the i-th group is denoted as n1, and the mean of the group is X1 where
iE[l,k]. The mean of all the samples is denoted as X0•
s;,
The between-groups mean sum ofsquares, is an estimate of the between-groups variance. It
measures how the population means vary with respect to the grand mean, or the mean spread across all
the populations. Formally, this is presented as shown in Equation 3-4.
k
582 =-1
-~n.·(x.-x0 )2
k-1L...i I I
(3-4)
1=1

The within-group mean sum ofsquares, s~. is an estimate ofthe within-group variance. It quantifies
the spread of values within groups. Formally, this is presented as shown in Equation 3-5.
REVIEW OF BASIC DATA ANALYTIC METHODS USING R

(3-5)

s;
If is much larger than 5~, then some of the population means are different from each other.
The F-test statistic is defined as the ratio of the between-groups mean sum of squares and the within-
group mean sum of squares. Formally, this is presented as shown in Equation 3-6.

(3-6)

The F-test statistic in ANOVA can be thought of as a measure of how different the means are relative to
the variability within each group. The larger the observed F-test statistic, the greater the likelihood that
the differences between the means are due to something other than chance alone. The F-test statistic
is used to test the hypothesis that the observed effects are not due to chance-that is, if the means are
significantly different from one another.
Consider an example that every customer who visits aretail website gets one oftwo promotional offers
or gets no promotion at all. The goal is to see if making the promotional offers makes a difference. ANOVA
could be used, and the null hypothesis is that neither promotion makes adifference. The code that follows
randomly generates atotal of 500 observations of purchase sizes on three different offer options.
offers<- sample(c("offerl", "offer2", "nopromo"), size=SOO, replace=T)

TRACE KTU
# Simulated 500 observations of purchase sizes on the 3 offer options
purchasesize <- ifelse(offers=="offerl", rnorm(SOO, mean=SO, sd=30),
ifelse(offers=="offer2", rnorm(SOO, mean=SS, sd=30),
rnorm(SOO, mean=40, sd=30)))

# create a data frame of offer option and purchase size

offertest <- [Link](offer=[Link](offers),
purchase_amt=purchasesize)

The summary ofthe offertest data frame shows that 170 offerl, 161 offer2, and 169
nopromo (no promotion) offers have been made. It also shows the range of purchase size (purchase_
amt) for each of the three offer options.
# display a summary of offertest where o:fer="offer1"
summary(offertest[offertest$offer=="offerl",])
offer purchase_amt
nopromo: t·li;;.. 4.521
offe:n :170 1 s:: Qu . : 5 8 . 1 5 8
offer2 : i·iedian : 76. 944
I·1ean .Sl. 936
3 rd Qu. : 1 D4 . 9 59
t•la:·:. :130.507

# display a summary of offertest where o:fer="offer2"

summary(offertest[offertest$offer=="offer2",])
 3.3 Statistical Methods for Evaluation

offer purchase_amt
nopromo: 0 ~lin. 14.04
offer! 0 1st Qu . : 6 9 . 46
offer2 :161 t·ledian : 90.20
r•lean 89.09
3 rd Qu. : 10 7. 4 8
!•lax. : 154. 3 3

# display a summary of offertest where offer="nopromo"

summary(offertest[offertest$offer== 11 nopromo 11 , ] )
offer purchase_amt
nopromo:169 Min. :-27.00
offerl 0 1st Qu.: 20.22
offer2 : 0 t<ledian : 42.44
[Link] 40.97
3rd Qu.: 58.96
i'olax. :164.04
The aov ( } function performs the ANOVA on purchase size and offer options.
# fit ANOVA test
model <- aov(purchase_amt - offers, data=offertest)

The summary (} function shows a summary of the model. The degrees of freedom for offers is 2,

TRACE KTU
which corresponds to the k -1 in the denominator of Equation 3-4. The degrees of freedom for residuals
is 497, which corresponds to then- k in the denominator of Equation 3-5.
summary (model)
Of Sum Sq !-lean Sq F value Pr (>F)
offers 2 225222 112611 130.6 <2e-16
Residuals 497 428470 862

Signif. codes: 0 1
*** 1
0.001 1
**' 0.01 1
* 1
0.05 '. 1
0.1 1 1
1

The output also includes the 5~ (112,611), 5~ (862), the F-test statistic (130.6), and the p-value (< 2e-16).
The F-test statistic is much greater than 1with ap-value much less than 1. Thus, the null hypothesis that
the means are equal should be rejected.
However, the result does not show whether offerl is different from offer2, which requires addi-
tional tests. The TukeyHSD (} function implements Tukey's Honest Significant Difference (HSD) on all
pair-wise tests for difference of means.
TukeyHSD(model)
Tukey multiple comparisons of means
95% family-wise confidence level

Fit: aoviformula purchase amt - offers, data offertest)

$offers
diff lwr upr p adj
offerl-nopromo 40.961437 33.4638483 48.45903 0.0000000
REVIEW OF BASIC DATA ANALYTIC METHODSUSING R

offer2 nopromo 48 . 120286 40 . 5189 4~6 55 . 72163 O. OOOOOCO

offer2--fferl 7 . 1"8849 -0 . 4315769 14 . 74928 o . r692895

The result includesp -values of pair-wise comparisons of the three offer options. The p-values for
of ferl- nopromo and of fer- nop romo are equal to 0, smaller than the significance level 0.05.
Thi s suggests that both of ferl and offer2 are significantly different from n opromo. Ap-value of
0.0692895 for off er2 against of fer 1 is greater than the significance level 0.05. This suggests that
of fer2 is not significantly different from offerl.
Because only the influence of one factor (offers) was executed, the presented ANOVA isknown asone-
way ANOVA. If the goal is to analyze two factors, such as offers and day of week, that would be a two-way
ANOVA [16]. 1f the goal isto model more than one outcome variable, then multivariate ANOVA (or MANOVA)
cou ld be used.

Summary
Ris a popular package and programming language for data exploration, analytics, and visualization. As an
introduction toR, thischapter coversthe RGUI, data 1/0, attribute and datatypes, and descriptive statistics.
This chapter also discusses how to useR to perform exploratory data analysis, including the discovery of
dirty data, visua lization of one or more variables, and customization of visualization for different audiences.
Finally, thechapter introduces some basic statistical methods. The first statistical method presented in the
chapter isthe hypothesis testing. The Student's t-test and Welch's t-test are included astwo examplehypoth-

TRACE KTU
esis testsdesigned for testing the difference of means. Other statistical methods and toolspresented in this
chapter include confidence interva ls, Wilcoxon rank-sum test, type I and II errors, effect size, and ANOVA.

Exercises
1. How many levels does fdata contain in the following Rcode?

data = c(1 , 2,2,3,1,2 , 3,3 ,1 , 2,3,3 , 1)

fdata = factor(data)

2. Two vectors, vl and v2, are created with the following Rcode:

vl <- 1:5
v2 <- 6 : 2

What are the results of cbi nd (vl , v2) and rbind (vl , v2)?
3. What Rcomma nd(s) would you use to remove null values from a dataset?

4. What Rcommand can be used to install an additional Rpackage?

5. What Rfunction isused to encode a vector as a category?

6. What is a rug plot used for in a density plot?

7. An online retailer wa nts to study the purchase behaviors of its customers. Figure 3-27 shows the den-
sity plot of the purchase sizes (in dollars).What wou ld be your recommendation to enhance the plot
to detect more structures that otherwise might be missed?
 Bibliography

Be-04

6e-04

£
"'
~ 4e-04

2e-04

09+(}0

0 2000 4000 6000 8000 10000

purchase size (dollars)

fiGURE 3-27 Density plot ofpurchase size

TRACE KTU
8. How many sections does a box-and-whisker divide the data into? What are these sections?

9. What attributes are correlated according to Figure 3-18? How would you describe their relationships?

10. What function can be used to tit a nonlinear line to the data?

11. If a graph of data is skewed and all the data is positive, what mathematical technique may be used to
help detect structures that might otherwise be overlooked?

12. What is a type I error? What is a type II error? Is one always more serious than the other? Why?

13. Suppose everyone who visits a retail website gets one promotional offer or no promotion at all. We
want to see if making a promotional offer makesa difference. What statistical method wou ld you
recommend for thisanalysis?

14. You are ana lyzing two norma lly distributed populations, and your null hypothesis is that the mean f1 1
of the first population is equal to the mean 112 of the second. Assume the significance level is set at
0.05. Ifthe observed p·value is 4.33e-05, what will be your decision regarding the null hypothesis?

Bibliography
[1] The RProject for Statistical Computing, "R Licenses." [Online). Available: http : I l www. r-
proj ec t. orgiLicensesl. [Accessed 10 December 2013].
[2] The RProject for Statistical Computing, "The Comprehensive RArch ive Network." [Online].
Available: http: I lcran . r-project. orgl. [Accessed 10 December 2013].