巨量資料探勘與應用

Big Data Mining and Applications

Python Data Science Modules I

李建樂
Chien-Yueh Lee, Ph.D.
Assistant Professor
Master Program in Artificial Intelligence
Innovation Frontier Institute of Research for Science and Technology
Department of Electrical Engineering
National Taipei University of Technology
Mar. 10, 2025
Outline
• Modules and Packages • Overview and Installation of
• Using Modules Third-party Packages
• Introduction to Built-in Modules • Querying Methods and Attributes
Provided by a Module
Ødecimal
Øitertools • NumPy
Østatistics
Øpickle
Module
• A module is a file that contains related code.
• Using modules helps improve code reusability and makes
maintenance easier.
• Python provides a rich set of built-in modules, including
mathematics, file handling, data compression, file formats,
encryption services, operating system services, concurrency,
network protocols, web data processing, multimedia,
graphical user interfaces, development tools.
• [Link]
Package
• A package is a collection of multiple
modules, similar to a folder.
• In principle, any directory containing an
__init__.py file is recognized as a Python
package.
• For example, the following layout
represents a package structure:
package
├── __init__.py
├── subpackage1
│ ├── __init__.py
│ ├── [Link]
│ └── [Link]
├── subpackage2
│ ├── __init__.py
│ └── [Link]
└── [Link]
Using a Module
• A module defines data, functions, or classes. To use its features, it must be imported
using the import statement. The syntax is as follows:
import moduleName
• You can assign an alias to a module with:
import moduleName as alias
• You can import specific functions or classes from a module using:
from moduleName import className1/functionName1, className2/functionName2, …
• You can import all functions and classes from a module using:
from moduleName import *
• For larger packages containing multiple modules, you may need to specify the
module explicitly using [Link]. The syntax is as follows:
from [Link] import className/functionName
Built-in Modules－decimal
• Use the decimal module to define a new Decimal class as a replacement for the float
type.
• The Decimal class precisely represents floating-point numbers.
• Unlike the float type, which uses binary representation and may have precision issues,
Decimal provides accurate decimal arithmetic.
• Before using the decimal module, it must be imported with the import statement.
import decimal

• [Link](v) - Create a new Decimal object from the input value.

• [Link](v).sqrt() - Return the square root of the value.
• [Link](v).exp() - Return the exponentiation of Euler's constant e.
• [Link](v).ln() - Return the natural logarithm of the value.
• [Link](v).log10() - Return the base-10 logarithm of the value.
The Pitfall of Floating-Point Numbers
• Why does the following code result in 0.1+0.2 ≠ 0.3?
 The Floating-Point Precision Issue
• When a computer stores floating-point numbers, it uses binary
representation. However, due to differences between binary and decimal
notation, precision issues arise.
• Floating-point numbers are stored in the computer as an exponent and a
mantissa:
Ø The exponent determines the position of the decimal point in the mantissa.
Ø The mantissa represents the actual numeric value using binary fractions, such as
1/2, 1/4, 1/8, etc.

Single-precision floating-point format

Sign Exponent Mantissa

Double-precision floating-point format

Sign Exponent Mantissa

 The Floating-Point Precision Issue
• Inexact Conversion of Floating-Point Numbers
ØSome decimal fractions cannot be precisely converted into a finite binary
representation. E.g., the decimal 0.1 in binary is an infinite repeating
fraction: 0.00011001100...
• Limited Storage Capacity
ØComputers store floating-point numbers using a fixed number of bits.
ØSince they cannot store infinite repeating fractions, they must
approximate the value within the available bits.
• Accumulated Errors in Calculations
ØThe results of floating-point operations gradually accumulate errors,
especially when multiple floating-point operations are performed.

Conclusion: 0.1 and 0.2 are approximations in binary representation. When added
together, the accumulated precision error prevents the exact calculation of 0.3.
Built-in Modules－decimal
 Built-in Modules－itertools
• The itertools module provides multiple efficient iterator functions. Before using the itertools
module, it must be imported using the import statement.
import itertools
• [Link](str) - Create a cyclic character iterator from a string str.
• [Link](x, n) - Repeat x for n times.
• [Link](L) - Compute the accumulated sum of a list L.
• [Link](str, n) - Generate permutations of n elements from a string str.
• [Link](str, n) - Generate combinations (without repetition) of n
elements from a string str
• itertools.combinations_with_replacement(str, n) - Generate combinations (with
repetition) of n elements from a string str.
• [Link](str1, str2) - Compute the Cartesian product of str1 and str2.
Built-in Modules－itertools
Built-in Modules－itertools
 Built-in Modules－statistics
• The statistics module provides basic statistical functions.
• For more advanced statistical functions, third-party libraries such as NumPy or SciPy should
be used.
• Before using the statistics module, it must be imported using the import statement.

import statistics

• [Link](L) - Calculate the mean (average)

• [Link](L) - Calculate the median
• statistics.median_low(L) - Calculate the lower median (for an even number of data points)
• statistics.median_high(L) - Calculate the upper median (for an even number of data points)
• [Link](L) - Calculate the mode (眾數)
• [Link](L) / [Link](L) - Calculate the sample (樣本) standard
deviation/variance
• [Link](L) / [Link](L) - Calculate the population (母體) standard
deviation/variance
Built-in Modules－statistics
Built-in Modules－pickle
• The pickle module provides binary serialization and deserialization
for object structures.
• Using pickle, you can store, transmit, and minimize variable data.
• For example:
Øpickle can save an instantiated object (實體化物件) to a file.
ØIt can package and share trained machine learning/deep learning models
with other users.
• Before using the pickle module, it must be imported using the import
statement.
import pickle
• [Link](x, file_obj) - Save as a pickle file.
• [Link](file_obj) - Read a pickle file.
Built-in Modules－pickle
Third-Party Packages
• Compared to built-in modules and packages that come pre-installed with Python, third-party
packages need to be installed separately. Common third-party packages include:
Ø NumPy: Array and Data Computation
Ø Matplotlib, Seaborn：2D Visualization Tools
Ø Bokeh, Plotly：Web Interactive Visualization Tools
Ø SciPy：Scientific Computing
Ø pandas：Data Processing and Analysis
Ø pySpark, Dask, Vaex, Ray, Modin, RAPIDS：Big Data Distributed Computing Frameworks
Ø Django, Flask, Pyramid, Web2py：Web Frameworks
Ø Kivy, Flexx, Pywin32, PyQt, WxPython：GUI Application Development
Ø BeautifulSoup：HTML/XML Parsers
Ø OpenCV, Pillow：Graphics Processing
Ø PyGame：Multimedia and Game Development
Ø Requests：Accessing Internet Data
Ø Scrapy：Web Scraping Packages
Ø SciKit-Learn, TensorFlow, Keras, PyTorch：Machine Learning and Deep Learning Packages
Installing Third-Party Packages with pip
• pip is a package management tool for Python that allows users to view, install,
upgrade, and remove packages.
• pip is not a Python statement and cannot be executed directly in a script.
Instead, it must be used in a command-line environment such as
Windows/macOS terminals or bash shell in Linux.
• Common pip Commands:
Ø pip list – Lists currently installed packages and their versions.
Ø pip install – Installs a package. For example, the following command installs the
NumPy package:
pip install numpy

Ø pip show – Displays information about an installed package.

Ø pip uninstall – Uninstalls a package. For example, the following command
removes the NumPy package:
pip uninstall numpy
Installing Third-Party Packages with pip
• In the Jupyter Notebook environment, to use pip commands, prefix
them with an exclamation mark !, indicating execution as an operating
system command.
Installing Third-Party Packages via PyPI
• The Python Package Index (PyPI) website ([Link] lists
tens of thousands of third-party packages.
• To install a package:
Ø Visit the PyPI website.
Ø Enter the package name (e.g., numpy) in the search bar.
Ø Download and install the package on your computer.
dir() Function
• Using the dir() function returns a list of variables, methods, and defined types in the
current scope.
>>> A = 123
>>> B = 'ABC'
>>> dir()
['A', 'B', '__annotations__', '__builtins__', '__doc__',
'__loader__', '__name__', '__package__', '__spec__']
• The dir() function, combined with the in operator, can be used to check whether a
specific variable exists.
>>> if 'A' in dir():
... print('The variable A exists.')
... else:
... print('The variable A does not exist.')
...
The variable A exists.
dir() Function
• If a parameter is provided, dir() returns a list of the parameter's
attributes and methods.
• The following example lists the methods and attributes provided by a
module:
>>> import math
>>> print(dir(math))
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'comb', 'copysign',
'cos', 'cosh', 'degrees', 'dist', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial',
'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite',
'isinf', 'isnan', 'isqrt', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf',
'nan', 'perm', 'pi', 'pow', 'prod', 'radians', 'remainder', 'sin', 'sinh', 'sqrt', 'tan',
'tanh', 'tau', 'trunc']
NumPy
• Creating Arrays
• Reading Arrays
• Attributes of ndarray Type
• Assigning and Modifying Arrays
• Searching for Array Elements
• Array, Vector, and Matrix Operations
• Shallow Copy vs. Deep Copy in Arrays
• Broadcasting
• Universal Function (ufunc)
• File Data Input/Output
Understanding NumPy
• NumPy (Numeric Python, pronounce [`nəmpaɪ]) is a third-party
package for scientific computing in Python.
• It supports array and matrix operations and includes a large library of
mathematical and statistical functions, enabling efficient scientific
computing and multidimensional data analysis.
• To install NumPy, use the following command:
pip install numpy
• Before using NumPy, the module must be loaded. In practice, it is
commonly imported with the alias np:
import numpy as np
Creating Arrays
• ndarray is an array data structure used to store multidimensional data. In
NumPy, most operations are performed using ndarray.
• The data stored in an array is called elements, representing values.
• NumPy provides several methods to create arrays:
Method Applicable Dimensions Description
[Link]() Multidimensional Converts a list into an array
Creates an array of a specified size with
[Link]() Multidimensional
uninitialized values
[Link]() Multidimensional Creates an array filled with zeros
[Link]() Multidimensional Creates an array filled with ones
Creates an identity matrix (diagonal elements =
[Link]() Multidimensional
1, others = 0)
Creates an array with evenly spaced values within
[Link]() One-dimensional
a range, given a step size
Creates an array with evenly spaced values within
[Link]() One-dimensional
a range, given a specific number of elements
NumPy Data Types
Type Description Type Description
bool Boolean, 0 or 1 float Floating point, equivalent to float64
Signed integers (positive and negative float16 Half-precision floating point
int
integers), equivalent to int64
float32 Single-precision floating point
int8 8-bit integer, -128～127
float64 Double-precision floating point
int16 16-bit integer, -32,768～32,767
Floating-point complex number,
complex
32-bit integer, -2,147,483,648～ equivalent to complex128
int32
2,147,483,647 Complex number type composed of 2
complex64
64 bit integer, 32-bit-precision floating-point numbers
int64 -9,223,372,036,854,775,808～ Complex number type composed of 2
complex128
9,223,372,036,854,775,807 64-bit-precision floating-point numbers
Unsigned integers (positive integers), object Object
uint
equivalent to uint64 byte Signed bit
uint8 8-bit unsigned integer, 0～255 ubyte Unsigned bit
uint16 16-bit unsigned integer, 0～65,535 unicode Unicode
uint32 32-bit unsigned integer, 0～4,294,967,295
64-bit unsigned integer, 0～
uint64
18,446,744,073,709,551,615
Converting List Data to an Array
• [Link]() can convert a given list into a NumPy array.
• The following examples create one-dimensional, two-dimensional, and
three-dimensional arrays, respectively.
Converting List Data to an Array
• When creating an array with [Link](), adding the dtype parameter
allows conversion of the original list data into a specified type.
Creating an Arbitrary-Value Array with a
Specified Size
• [Link]() creates an array of a specified size in an uninitialized
state, meaning its elements contain arbitrary values.
• Note: [Link]() does not create an empty array as its name might
suggest. If you need an array filled with zeros, use [Link]() instead.
Other Methods for Creating Arrays
• [Link]() - Creates an array where every element is 0.
• [Link]() - Creates an array where every element is 1.
• [Link]() - Creates an identity matrix where the diagonal elements are 1,
and all other elements are 0.
Other Methods for Creating Arrays
• [Link]() - Creates an array with evenly spaced values between two
numbers, specifying the step size.
• Note: The endpoint is not included.
Other Methods for Creating Arrays
• [Link]() - Creates an array with evenly spaced values between two
numbers, specifying the number of elements.
• Note: The endpoint is included.
Data Type Conversion
• Use the astype() method to change the data type of array elements.
Reading Arrays
• NumPy provides several methods to read array elements:
1. Indexing: Access specific elements using their index values.
2. Slicing: Use slice notation (similar to lists) to extract a range
of elements.
3. Masking: Use a boolean mask array to filter elements where
the condition evaluates to True.
Indexing in 1D NumPy Arrays
• For a one-dimensional ndarray array, you can access an element by
specifying its index.
ØIndex 0 represents the first element.
ØIndex 1 represents the second element, and so on.
ØIndex n - 1 represents the nth element in the array.
Element Value
 Indexing in 2D NumPy Arrays
• For a two-dimensional array, you can:
ØUse an index list to access multiple elements.
ØProvide two index values (row index, column index) to retrieve a specific
element, similar to accessing elements in a 2D list.
Chinese English Math
Student 1
Student 2
Student 3
Student 4
Student 5

Chinese English Math

Student 1
Student 2
Student 3
Student 4
Student 5
Slicing in NumPy Arrays
• Similar to list operations, an ndarray array can use slice notation to
extract a range of elements, creating a new array.
Masking in NumPy Arrays
• A mask array in NumPy is created by applying a condition to each element in
the target array. This results in a Boolean array of the same shape, where each
element is True or False.
• Using the mask array, you can filter the target array, selecting only the
elements where the mask is True.
1 2 3 False True False 1
False 2
True 3
False
N % 2 == 0 Filter
4 5 6 True False True 4
True 5
False 6
True

7 8 9 False True False 7

False 8
True 9
False

Output

2 4 6 8
Masking in NumPy Arrays
Attributes of the ndarray Type
• Key attributes of an ndarray in NumPy:
Ø[Link] - Number of dimensions in the array
Ø[Link] - Shape of the array (dimensions)
Ø[Link] - Total number of elements in the array
Ø[Link] - Data type of array elements
Ø[Link] - Size (in bytes) of each element in
the array
Assigning and Modifying Arrays
• Assignment in NumPy arrays works similarly to lists. Once you know
the position of an element in the array, you can assign a new value to it.
Assigning and Modifying Arrays
• NumPy arrays support bulk assignment, allowing the same value to be
assigned to multiple elements at once.

• Arithmetic operators and comparison operators Consider this: what happens

are applied element-wise to all elements in the array. if it is a list?
 Other Array Operations
• [Link]() - Combines two arrays or adds • [Link](arr, x, y) - Limits the values in arr
elements. to be within the range [x, y].

• [Link]() - Inserts elements into an array.

• [Link]() - Deletes elements from an array.

Modifying an Array Shape
• reshape() - Converts an existing array into a specified shape.

• [Link]() or flatten() - Flattens a multidimensional array into a one-

dimensional array.
Searching for Array Elements
• [Link]() - Finds indices based on a condition and returns an array of
indices.

(0, 3) → 4
(1, 0) → 5
(1, 1) → 6
(1, 2) → 7
(1, 3) → 8
Array Operations
• Common ndarray operations:
Øsum([axis]) - Computes the sum
Ømean([axis]) - Computes the mean (average)
Øvar([axis]) - Computes the variance axis=1

Østd([axis]) - Computes the standard deviation

Ømin([axis]) - Finds the minimum value
Ømax([axis]) - Finds the maximum value
axis=0
Vector Operations
• Inner (內積), cross (叉積), and outer products (外積)
Ø[Link]() - Computes the inner product of two vectors.
Ø[Link]() - Computes the cross product of two vectors.
Ø[Link]() - Computes the outer product of two vectors.
Matrix Operations
• [Link]() or .T - Transposes a matrix.
Matrix Operations
• + (Addition Operator) - Performs
matrix addition.
• @ (Matrix Multiplication Operator)
or [Link]() - Performs matrix
multiplication.
Shallow Copy vs. Deep Copy in Arrays
• A simple assignment does not create a copy of the array's data, e.g.,

← A and B reference the same object.

• A shallow copy (called a view) creates an index on the original array.

Changes in the original array will also reflect in the new array.

← A and C are different objects.

← C is a view of A and shares A's data.
Shallow Copy vs. Deep Copy in Arrays
• A deep copy (called a copy) duplicates the original array's contents
completely. The new array and the original array are independent and
do not affect each other.

← A and D are different objects.

← D is a copy of A and has independent data.
Broadcasting
• In principle, two arrays must have compatible shapes for arithmetic operations.
• If their shapes differ, the smaller array is expanded according to NumPy's
broadcasting mechanism to match the shape of the larger array.
• Examples:
Ø A is a 1D array, and B is a scalar. Before performing A + B, B is expanded to [10, 10, 10], so
the result is [11, 12, 13].

Ø C is a (2, 3) 2D array. Before performing A + C, A is expanded to [[1, 2, 3], [1, 2, 3]] to match
C's shape before computation.
 Universal Function (ufunc)
• NumPy provides common mathematical functions such as sin(), cos(),
exp(), square(), and add(), which are known as universal functions (ufuncs).
• These functions operate element-wise on an array.
• The result is a new array with the computed values.
>>> import numpy as np
>>> A = [Link]([1, 2, 3])
>>> B = [Link](A)
>>> B
array([1, 4, 9], dtype=int32)
 Mathematical Functions
＊represents universal functions
• Trigonometric Functions (＊): cos(x), sin(x), tan(x), acos(x), asin(x), atan(x)
• Rounding Functions (＊): round(x[, decimals=0]), rint(x), floor(x), ceil(x), trunc(x)
• Sum/Product/Difference Operations: prod(a[, axis=None]), cumprod(a[, axis=None]),
sum(a[, axis=None]), cumsum(a[, axis=None]), diff(a[, n=1, axis=-1]), cross(a, b)
• Exponential and Logarithmic Functions (＊): exp(x), exp2(x), log(x), log2(x), log10(x)
• Arithmetic Operations (＊): add(x1, x2), subtract(x1, x2), multiply(x1, x2), divide(x1, x2),
power(x1, x2), mod(x1, x2), remainder(x1, x2), fmod(x1, x2), divmod(x1, x2), negative(x)
• Other Functions (＊): sign(x), absolute(x), sqrt(x), cbrt(x), square(x), maximum(x1, x2),
minimum(x1, x2), gcd(x1, x2)
• Other Useful Functions: isinf(x), isfinite(x), isnan(x), max(x), min(x), sort(x)
Random Sampling Functions
• [Link].X • Shuffling Order
• Simple Random Data § shuffle(x) - Shuffle the elements of array
x in place.
§ rand(d0, d1, …, dn)
§ permutation(x) - Return a new array
§ randn(d0, d1, …, dn)
with the elements of x reordered.
§ randint(low[, high, size, dtype])
• Random Number Generators
§ random_integers(low[, high, size])
§ seed(seed=None)
§ random_sample([size])
§ random([size])
§ ranf([size])
§ sample([size])
§ choice(a[, size, replace, p])
Random Sampling Functions
§ multivariate_normal(mean, cov[, size, …)
• [Link].Y § negative_binomial(n, p[, size])
• Distribution (分佈) § noncentral_chisquare(df, nonc[, size])
§ noncentral_f(dfnum, dfden, nonc[, size])
§ beta(a, b[, size])
§ normal([loc, scale, size])
§ binomial(n, p[, size])
§ pareto(a[, size])
§ chisquare(df[, size])
§ poisson([lam, size])
§ dirichlet(alpha[, size])
§ power(a[, size])
§ exponential([scale, size])
§ rayleigh([scale, size])
§ f(dfnum, dfden[, size])
§ standard_cauchy([size])
§ gamma(shape[, scale, size])
§ standard_exponential([size])
§ geometric(p[, size]) § standard_gamma(shape[, size])
§ gumbel([loc, scale, size])
§ standard_normal([size])
§ hypergeometric(ngood, nbad, § standard_t(df[, size])
nsample[, size])
§ laplace([loc, scale, size]) § triangular(left, mode, right[, size])
§ logistic([loc, scale, size]) § uniform([low, high, size])
§ lognormal([mean, sigma, § vonmises(mu, kappa[, size])
size]) § wald(mean, scale[, size])
§ logseries(p[, size]) § weibull(a[, size])
§ multinomial(n, pvals[, size]) § zipf(a[, size])
Statistical Functions
• amin(a, axis=None)
• amax(a, axis=None)
• nanmin(a, axis=None)
• nanmax(a, axis=None)
• average(a, axis=None, weights=None)
• median(a, axis=None)
• mean(a, axis=None)
• std(a, axis=None)
• var(a, axis=None)
• nanmedian(a, axis=None)
• nanmean(a, axis=None)
• nanstd(a, axis=None)
• nanvar(a, axis=None)
File Data Input/Output
• You can use [Link]() to read the text file [Link] into a NumPy array.

>>> [Link]("[Link]", delimiter = ',')

array([[ 15., 160., 48.],
[ 14., 175., 66.],
[ 15., 153., 50.],
[ 15., 162., 44.]])
>>> [Link]("[Link]", delimiter = ',', skiprows = 2) # Skip the first 2 rows
array([[ 15., 153., 50.],
[ 15., 162., 44.]])
File Data Input/Output
• The following code creates three arrays (x, y, z) and writes them to a
text file [Link].
• The format string "%1.2f" ensures that each floating-point number is
printed with at least one digit before the decimal point and two
decimal places.
>>> x = y = z = [Link](0, 5, 1)
>>> [Link]("[Link]", (x, y, z), delimiter = ',', fmt = "%1.2f")
HW3-1
• Create a mask array to replace even elements in the
following NumPy array with -1
arr = [Link]([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
 HW3-2
• Given the admission scores of a music school, use NumPy to answer the following
questions:
Major Minor Singing Music Theory Dictation
Student 1 80 75 88 80 78
Student 2 88 86 90 95 86
Student 3 92 85 92 98 90
Student 4 81 88 80 82 85
Student 5 75 80 78 80 70

a. Print the average scores of each student.

b. Given the weights of the five subjects (50%, 20%, 10%, 10%, 10%), print the weighted
average scores of each student.
c. Print the median, standard deviation, and variance of each student's scores.
HW3-3
• Solve the Following Problems Using NumPy:
a. Generate an array A with six numbers (1, 3, 5, 7, 9, 11) using
[Link]() and store them in a 3×2 matrix.
b. Print the dimensions, shape, and total number of elements in array A.
c. Print the maximum and minimum elements in array A.
d. Define array B as the transpose of A, then print B.
e. Define array C as the sum of A + A, then print C.
f. Define array D as the matrix product of A and B, then print D.
HW3-4
• Please use NumPy slicing and bulk assignment to generate
the following square matrix.